Multi-scale Contextual Information Research Articles

CBFNet: Constraint balance factor for semantic segmentation

The deep neural network model based on spatial pyramid pooling for multi-scale context information has been widely adopted in image semantic segmentation. However, this structure reduces the feature resolution during the inference process, which should be recovered by bilinear interpolation. We find that the performance of boundary location heavily depends on the interpolation operation. Assuming that the foreground object has a boundary with uneven curvature changes, it will be difficult to preserve details in the lower resolution feature map. In this case, if the model achieves near-optimal fitting, the prediction of the boundary point needs to satisfy strict constraints. But this condition is very difficult to meet for the model. For this purpose, we propose a constraint balance factor (CBF) relaxation method to improve the fitting ability of the model. The constraint balance factor is learned end-to-end by a sub-network. This sub-network can be easily embedded into a semantic segmentation network that requires interpolation without too many extra parameter increments. The experimental results illustrate that the proposed CBFNet performs well on PASCAL VOC 2012 and Cityscapes datasets.

Densely connected network for impulse noise removal

Recently, a new convolutional neural network (CNN) architecture, dubbed as densely connected convolutional network (DenseNet), has shown excellent results on image classification tasks. The idea of DenseNet is based on the observation that if each layer is directly connected to every other layer in a feed-forward fashion, then the network will be more accurate and easier to train. In this study, we extend DenseNet to deal with the problem of impulse noise reduction. It aims to explore the densely connected network for impulse noise removal (DNINR), which utilizes CNN to learn pixel-distribution features from noisy images. Compared with the traditional median filter-based and variational regularization methods that utilize the spatial neighbor information around the pixels and optimize in an iterative manner, it is more efficient to capture multi-scale contextual information and directly tackles the original image. Additionally, DNINR turns to capture the pixel-level distribution information by means of wide and transformed network learning. In terms of edge preservation and noise suppression, the proposed DNINR consistently achieved significantly superior performance, which is better than current state-of-the-art methods, particularly at extremely high noise levels.

Fusing multi-scale context-aware information representation for automatic in-field pest detection and recognition

Automatic in-field pest detection and recognition using mobile vision technique is a hot topic in modern intelligent agriculture, but suffers from serious challenges including complexity of wild environment, detection of tiny size pest and classification of multiple classes of pests. While recent deep learning based mobile vision techniques have shown some success in overcoming above issues, one key problem is that towards large-scale multiple species of pest data, imbalanced classes significantly reduce their detection and recognition accuracy. In this paper, we propose a novel two-stages mobile vision based cascading pest detection approach (DeepPest) towards large-scale multiple species of pest data. This approach firstly extracts multi-scale contextual information of the images as prior knowledge to build up a context-aware attention network for initial classification of pest images into crop categories. Then, a multi-projection pest detection model (MDM) is proposed and trained by crop-related pest images. The role of MDM can combine pest contextual information from low-level convolutional layers with these in high-level convolutional layers for generating the super-resolved feature. Finally, we utilize the attention mechanism and data augmentation to improve the effectiveness of in-field pest detection. We evaluate our method on our newly established large-scale dataset In-Field Pest in Food Crop (IPFC) and sufficient experimental results show that DeepPest proposed in this paper outperforms state-of-the-art object detection methods in detecting in-field pest.

Deep Learning for The Automatic Diagnosis and Detection of Multiple Retinal Abnormalities

This paper presents a fully-automated method to detect retinal abnormalities in both a global and local context by generating: (1) a disease label with a probability score and (2) a 4-channel pixel-level segmentation map of retinal lesions. The characteristics of retinal abnormalities, which occur as various shapes, sizes, and distribution at different regions, are a challenge in accomplishing these tasks. In addition, the small amount of image-level labelled images in public databases and the unavailability of lesion-level annotations for most of these publicly available images also pose as challenges. These shortcomings motivate our exploration of various CNN architectures to extract multi-scale contextual information, such that we investigate the impact of different arrangements of multi-sized convolutional kernels appended to a modified pre-trained encoder. Additionally, to prevent the loss of detailed information for small lesions, we exploit the advantages of feature map concatenation from the output of these multi-scale convolutions to its corresponding decoder layer. A new two-phase training strategy is also implemented to tackle the problem of dataset imbalance between image-level label and lesion-level label classes. The direct comparison between our proposed methods and currently published state-of-the-art methods with the same databases confirms that our best model outperforms existing published methods.

Arbitrary Shape Natural Scene Text Detection Method Based on Soft Attention Mechanism and Dilated Convolution

Natural scene text detection has attracted much attention in the research field of computer vision, and it has been widely used in many applications, i.e., unmanned driving, robot sensing. Some methods have been proposed for horizontal and oriented text detection, but detecting irregular shapes and highly varying orientated text is still a challenging problem. To tackle this problem, we propose a robust arbitrary shape text detection method called Soft Dilated network (SDnet). The proposed method has two essential steps: (1) feature extraction by backbone; (2) post-processing approach to generate elaborated polygons or boundaries. In particular, the backbone is based on soft attention mechanism and dilated convolution. The soft attention mechanism learns and obtains importance feature from each feature channel, and dilated convolution can effectively aggregate multi-scale contextual information without losing the resolution, and enhance the robust of the network model. The proposed method can accurately detect curve text and discriminate text and non-text areas in an efficient fashion. In addition, Jaccard coefficient is used as loss function to promote the post-processing capability of detecting sparse-arranged and arbitrary shape text. Based on the aforementioned technique, the proposed method an effectively handle the problem of sparse arranged arbitrary natural scene text detection. Experiments were conducted on three benchmark datasets: curved text dataset CTW1500, Total-Text and oriented dataset ICDAR2015, and the results show that when compared with the state-of-the-art text detection methods, the proposed method is more robust and it can find smaller text blocks in the image due to the Loss Function calculation with Jaccard coefficient. Furthermore, we performed multiple sets of ablation experiments, verify the effectiveness of the propose method.

Accurate Pixel-Wise Skin Segmentation Using Shallow Fully Convolutional Neural Network

Skin segmentation plays an important role in human activity recognition, video surveillance, hand gesture identification, face detection, human tracking and robotic surgery. The accurate segmentation of the skin is necessary to recognize the human activity. Segmentation of skin is easy to realize in ideal situations because of similar backgrounds. But it becomes complicated because of presence of skin-like pixels, background illuminations, and certain changes in environment. These problems are addressed by incorporating preprocessing stages in current studies, but this raises the total cost of the system. However, there are some limitations associated with these methods in terms of accuracy and processing speed. In this work, we propose a skin semantic segmentation network (SSS-Net) that is able to capture the multi-scale contextual information and refines the segmentation results especially along object boundaries. Moreover our network helps to reduce the cost of the preprocessing as well. We have performed experiments on the five open datasets of human activity recognition for the segmentation of skin. Experimental results show SSS-Net outperforms the state-of-the-art methods in skin segmentation in terms of accuracies.

LDPNet: A Lightweight Densely Connected Pyramid Network for Real-Time Semantic Segmentation

A deep convolutional neural network has been widely used in image semantic segmentation in recent years, its deployment on mobile terminals, however is limited by its high computational costs. Given the slow inference speed and large memory usage of deep convolutional neural networks, we propose a lightweight and densely connected pyramid network (LDPNet) for real-time semantic segmentation. Firstly, a densely connected atrous pyramid (DCAP) module is constructed in the encoding process to extract multi-scale context information for forwarding propagation, strengthen the reuse of features, and offset the spatial information lost in the down-sampling process of the feature map. Secondly, a cross-fusion (CF) module embedded in each other during the decoding process is proposed, which uses high-level semantic features to effectively guide the fusion of low-level spatial details while strengthening context information. Our network is tested on two complex urban road scene data sets. Among them, experiments on the Cityscapes data set show that our structure has 87 frames per second (FPS) on a single NVIDIA GTX1080Ti GPU. The Mean Intersection over Union (mIoU) reaches 71.1%, and the parameter is only 0.8M. Compared with the existing similar networks, the new system achieves a state-of-the-art trade-off between efficiency and accuracy.

Semantic segmentation with DenseNets for carotid artery ultrasound plaque segmentation and CIMT estimation.

The measurement of carotid intima media thickness (CIMT) in ultrasound images can be used to detect the presence of atherosclerotic plaques. Usually, the CIMT estimation strategy is semi-automatic, since it requires: (1) a manual examination of the ultrasound image for the localization of a region of interest (ROI), a fast and useful operation when only a small number of images need to be measured; and (2) an automatic delineation of the CIM region within the ROI. The existing efforts for automating the process have replicated the same two-step structure, resulting in two consecutive independent approaches. In this work, we propose a fully automatic single-step approach based on semantic segmentation that allows us to segment the plaque and to estimate the CIMT in a fast and useful manner for large data sets of images. Our single-step approach is based on densely connected convolutional neural networks (DenseNets) for semantic segmentation of the whole image. It has two remarkable characteristics: (1) it avoids ROI definition, and (2) it captures multi-scale contextual information in the complete image interpretation, due to the concatenation of feature maps carried out in DenseNets. Once the input image is segmented, a straightforward method for CIMT estimation and plaque detection is applied. The proposed method has been validated with a large data set (REGICOR) of more than 8000 images, corresponding to two territories of the carotid artery: common carotid artery (CCA) and bulb. Among them, a subset of 331 images has been used to evaluate the performance of semantic segmentation (≈90% for train, ≈10% for test). The experimental results demonstrated that our method outperforms other deep models and shallow approaches found in the literature. In particular, our CIMT estimation reaches a correlation coefficient of 0.81, and a CIMT mean error of 0.02 and 0.06 mm in CCA and Bulb images, respectively. Furthermore, the accuracy for plaque detection is 96.45% and 78.09% in CCA and Bulb, respectively. To test the generalization power, the method has also been tested with another data set (NEFRONA) that includes images acquired with different equipment. The validation carried out demonstrates that the proposed method is accurate and objective for both plaque detection and CIMT measurement. Moreover, the robustness and generalization capacity of the method have been proven with two different data sets.

DW-Net: A cascaded convolutional neural network for apical four-chamber view segmentation in fetal echocardiography

Fetal echocardiography (FE) is a widely used medical examination for early diagnosis of congenital heart disease (CHD). The apical four-chamber view (A4C) is an important view among early FE images. Accurate segmentation of crucial anatomical structures in the A4C view is a useful and important step for early diagnosis and timely treatment of CHDs. However, it is a challenging task due to several unfavorable factors: (a) artifacts and speckle noise produced by ultrasound imaging. (b) category confusion caused by the similarity of anatomical structures and variations of scanning angles. (c) missing boundaries. In this paper, we propose an end-to-end DW-Net for accurate segmentation of seven important anatomical structures in the A4C view. The network comprises two components: 1) a Dilated Convolutional Chain (DCC) for “gridding issue” reduction, multi-scale contextual information aggregation and accurate localization of cardiac chambers. 2) a W-Net for gaining more precise boundaries and yielding refined segmentation results. Extensive experiments of the proposed method on a dataset of 895 A4C views have demonstrated that DW-Net can achieve good segmentation results, including the Dice Similarity Coefficient (DSC) of 0.827, the Pixel Accuracy (PA) of 0.933, the AUC of 0.990 and it substantially outperformed some well-known segmentation methods. Our work was highly valued by experienced clinicians. The accurate and automatic segmentation of the A4C view using the proposed DW-Net can benefit further extractions of useful clinical indicators in early FE and improve the prenatal diagnostic accuracy and efficiency of CHDs.

Dynamic attention network for semantic segmentation

Semantic segmentation networks usually utilize special pyramid structure after encoder or combine low-level and high-level feature maps in decoder to capture multi-scale context information, which we term them feature combination and feature connection, respectively. However, both such frameworks are less valid with the fixed geometric structure or unsuitable interim. In this paper, we advocate Dynamic Attention Network (DAN) to solve these problems. First, we design a Deformable Attention Pyramid (DAP) module to perform a self-adjustable descriptor of high-level output, which utilizes deformable function to model geometric transformation. With DAP, semantic information can be captured effectively. Second, we propose a Fusing Attention Interim (FAI) module to guide the back-propagation of long-short range information in each level of decoder. We evaluate DAN on the challenging PASCAL VOC 2012 and Cityscapes segmentation benchmarks and find that it achieves state-of-the-art results without post-processing. Our observation can be concluded that the flexible structure that possesses dynamic attention mechanism is beneficial to learn multi-scale context information.

Filling the Gaps in Atrous Convolution: Semantic Segmentation With a Better Context

The main challenge for scene parsing arises when complex scenes with highly diverse objects are encountered. The objects not only differ in scale and appearance but also in semantics. Previous works focus on encoding the multi-scale contextual information (via pooling or atrous convolutions) generally on top of compact high-level features (i.e., at a single stage). In this work, we argue that a rich set of cues exist at multiple stages of the network, encapsulating low, mid and high-level scene details. Therefore, an optimal scene parsing model must aggregate multi-scale context at all three levels of the feature hierarchy; a capability that lacks in state-of-the-art scene parsing models. To address this limitation, we introduce a novel architecture with three new blocks that systematically aggregate low, mid and high tier features. The heart of our approach is a high-level feature aggregation module that augments sparsely connected atrous convolution with dense local and layer-wise connections to avoid gridding artifacts. Besides, we employ a novel feature pyramid augmentation and semantic refinement unit to generate low- and mid-level features that are mixed with high-level features at the decoder. We extensively evaluate our proposed approach on the large-scale Cityscapes and ADE2K benchmarks. Our approach surpasses many latest models on both datasets, achieving mean intersection-over-union (mIoU) scores of 80.5% and 44.0% on Cityscapes and ADE20K, respectively.

Multi-scale dilated convolution of convolutional neural network for crowd counting

Growing numbers of crowd density estimation methods have been developed in scene monitoring, crowd safety and on-site management scheduling. We proposed a method for density estimation of a single static image based on convolutional neural network naming Multi-scale Dilated Convolution of Convolutional Neural Network (Multi-scale-CNN). The proposed method employed the method of density maps regression to learn the mapping relationship between single-image and density maps through convolutional neural network. The adopted network structure is composed of two major components to adapt changes of characters scales in crowd images, a convolutional neural network for the general feature extraction and the other is multi-scale dilated convolution for disposing the scale change problem. It is insufficient for currently study that tackled the multi-column or multi-input convolutional neural networks to solve multi-scale problems. Our method utilizes a single-column network to extract features and combines multi-scale dilated convolution to aggregate multi-scale information to address the shortcomings of two networks. The multi-scale dilated convolution module aggregates multi-scale context information systematically by making use of dilated convolution without reducing the receiving domain, thereby integrate the underlying detail information into the high-level semantic features to promote the perception and counting ability of network for small targets. This paper demonstrates the proposed network structure in ShanghaiTech dataset, UCF_CC_50 dataset and worldexpo’10 dataset, and compares the results with numbers of current mainstream crowd counting algorithms, proves that our method surpasses current state-of-the-art methods and has excellent counting accuracy and robustness. The training and testing codes of our method models can be downloaded at https://github.com/doctorwgd/Multi-scale-CNN.

Saliency detection using Multi-layer graph ranking and combined neural networks

In this paper, a new algorithm based on a combined neural network is proposed to improve salient object detection in the complex images. It consists of two main steps. The first step, an objective function which is optimized on a multi-layer graph structure is constructed to diffuse saliency from borders to salient objects, aiming to roughly estimate the location and extent salient objects of an image, meanwhile, color attribute is adopted to rapidly find a set of object-related regions in the image. The second step, establish a combined neural network with Region Net and Local-Global Net. Region Net is adopted to efficiently generate the salient map with the sharp object boundary. Then Local-Global Net based on multi-scale spatial context is proposed to provide strongly reliable multi-scale contextual information, and thus achieves an optimized performance. Experimental results and comparison analysis demonstrate that the proposed algorithm is more effective and superior than most low-level oriented prior methods in terms of precision recall curves, F-measure and mean absolute errors.

BowNet: Dilated convolutional neural network for ultrasound tongue contour extraction

One usage of medical ultrasound imaging is to visualize and characterize human tongue shape and motion during a real-time speech to study healthy or impaired speech production. Due to the low-contrast characteristic and noisy nature of ultrasound images, it might require expertise for non-expert users to recognize tongue gestures in applications such as visual training of a second language. Several end-to-end deep learning segmentation methods provide promising alternatives with higher accuracy and robustness results and without any intervention. Employing the power of the graphics processing unit with state-of-the-art deep neural network models makes it feasible to have new fully automatic, accurate, and robust segmentation methods with the capability of real-time performance. This paper presents a new novel deep neural network for tongue contour extraction, BowNet, benefits from exploitation capability of dilated convolution by effectively expanding the receptive field without losing resolution to extract clear tongue contours. Also, efficient abstract context exploration is carried out by down-sampling layers to achieve segmentation results with high resolution and relevancy. Two versions, BowNet and wBowNet, are studied qualitatively and quantitatively over datasets from two different ultrasound machines. Our experiment disclosed the outstanding performances of the proposed models in terms of accuracy and robustness in comparison with similar sized models.

Siamese Dilated Inception Hashing With Intra-Group Correlation Enhancement for Image Retrieval

For large-scale image retrieval, hashing has been extensively explored in approximate nearest neighbor search methods due to its low storage and high computational efficiency. With the development of deep learning, deep hashing methods have made great progress in image retrieval. Most existing deep hashing methods cannot fully consider the intra-group correlation of hash codes, which leads to the correlation decrease problem of similar hash codes and ultimately affects the retrieval results. In this article, we propose an end-to-end siamese dilated inception hashing (SDIH) method that takes full advantage of multi-scale contextual information and category-level semantics to enhance the intra-group correlation of hash codes for hash codes learning. First, a novel siamese inception dilated network architecture is presented to generate hash codes with the intra-group correlation enhancement by exploiting multi-scale contextual information and category-level semantics simultaneously. Second, we propose a new regularized term, which can force the continuous values to approximate discrete values in hash codes learning and eventually reduces the discrepancy between the Hamming distance and the Euclidean distance. Finally, experimental results in five public data sets demonstrate that SDIH can outperform other state-of-the-art hashing algorithms.

A Novel Deep Structure U-Net for Sea-Land Segmentation in Remote Sensing Images

Sea-land segmentation is an important process for many key applications in remote sensing. Proper operative sea-land segmentation for remote sensing images remains a challenging issue due to complex and diverse transition between sea and lands. Although several Convolutional Neural Networks (CNNs) have been developed for sea-land segmentation, the performance of these CNNs is far from the expected target. This paper presents a novel deep neural network structure for pixel-wise sea-land segmentation, a Residual Dense U-Net (RDU-Net), in complex and high-density remote sensing images. RDU-Net is a combination of both down-sampling and up-sampling paths to achieve satisfactory results. In each down- and up-sampling path, in addition to the convolution layers, several densely connected residual network blocks are proposed to systematically aggregate multi-scale contextual information. Each dense network block contains multilevel convolution layers, short-range connections and an identity mapping connection which facilitates features re-use in the network and makes full use of the hierarchical features from the original images. These proposed blocks have a certain number of connections that are designed with shorter distance backpropagation between the layers and can significantly improve segmentation results whilst minimizing computational costs. We have performed extensive experiments on two real datasets Google Earth and ISPRS and compare the proposed RDUNet against several variations of Dense Networks. The experimental results show that RDUNet outperforms the other state-of-the-art approaches on the sea-land segmentation tasks.

Random Shaped Image Inpainting using Dilated Convolution

Over the past few years, Deep learning-based methods have shown encouraging and inspiring results for one of the most complex tasks of computer vision and image processing; Image Inpainting. The difficulty of image inpainting is derived from its’ need to fully and deeply understand of the structure and texture of images for producing accurate and visibly plausible results especially for the cases of inpainting a relatively larger region. Deep learning methods usually employ convolution neural network (CNN) for processing and analyzing images using filters that consider all image pixels as valid ones and usually use the mean value to substitute the missing pixels. This result in artifacts and blurry inpainted regions inconsistent with the rest of the image. In this paper, a new novel-based method is proposed for image inpainting of random-shaped missing regions with variable size and arbitrary locations across the image. We employed the use of dilated convolutions for composing multiscale context information without any loss in resolution as well as including a modification mask step after each convolution operation. The proposed method also includes a global discriminator that also considers the scale of patches as well as the whole image. The global discriminator is responsible for capturing local continuity of images texture as well as the overall global images’ features. The performance of the proposed method is evaluated using two datasets (Places2 and Paris Street View). Also, a comparison with the recent state-of-the-art is preformed to demonstrate and prove the effectiveness of our model in both qualitative and quantitative evaluations.

CDnet: CNN-Based Cloud Detection for Remote Sensing Imagery

Cloud detection is one of the important tasks for remote sensing image (RSI) preprocessing. In this paper, we utilize the thumbnail (i.e., preview image) of RSI, which contains the information of original multispectral or panchromatic imagery, to extract cloud mask efficiently. Compared with detection cloud mask from original RSI, it is more challenging to detect cloud mask using thumbnails due to the loss of resolution and spectrum information. To tackle this problem, we propose a cloud detection neural network (CDnet) with an encoder–decoder structure, a feature pyramid module (FPM), and a boundary refinement (BR) block. The FPM extracts the multiscale contextual information without the loss of resolution and coverage; the BR block refines object boundaries; and the encoder–decoder structure gradually recovers segmentation results with the same size as input image. Experimental results on the ZY-3 satellite thumbnails cloud cover validation data set and two other validation data sets (GF-1 WFV Cloud and Cloud Shadow Cover Validation Data and Landsat-8 Cloud Cover Assessment Validation Data) demonstrate that the proposed method achieves accurate detection accuracy and outperforms several state-of-the-art methods.

Saliency detection via multi-level integration and multi-scale fusion neural networks

The recent advance on saliency models has remarkably improved performance due to the pervasive application of deep convolutional neural networks. However, for more challenging images, it is worthwhile to explore in deep convolutional neural networks how to effectively exploit features at different levels and scales for saliency detection. In this paper, we propose an end-to-end multi-level feature integration and multi-scale feature fusion network to better predict salient objects in challenging images. Specifically, our network first integrates multi-level features from high-level to low-level features in the network based on ResNet. Then, the feature combined by the multi-level feature integration network is further refined by four parallel residual connected blocks with dilated convolution, in which each block has a specific dilation rate to capture multi-scale context information. Finally, we fuse the outputs of residual connected blocks with dilated convolution and obtain the saliency map by up-sampling operation. Extensive experimental results demonstrate that the proposed model outperforms the state-of-the-art saliency models on several challenging image datasets.

Automatic Bridge Crack Detection Using a Convolutional Neural Network

Concrete bridge crack detection is critical to guaranteeing transportation safety. The introduction of deep learning technology makes it possible to automatically and accurately detect cracks in bridges. We proposed an end-to-end crack detection model based on the convolutional neural network (CNN), taking the advantage of atrous convolution, Atrous Spatial Pyramid Pooling (ASPP) module and depthwise separable convolution. The atrous convolution obtains a larger receptive field without reducing the resolution. The ASPP module enables the network to extract multi-scale context information, while the depthwise separable convolution reduces computational complexity. The proposed model achieved a detection accuracy of 96.37% without pre-training. Experiments showed that, compared with traditional classification models, the proposed model has a better performance. Besides, the proposed model can be embedded in any convolutional network as an effective feature extraction structure.