Multi-scale Objects Research Articles

HCNet: Hierarchical Context Network for Semantic Segmentation

Global context information is vital in visual understanding problems, especially in pixel-level semantic segmentation. The mainstream methods adopt the self-attention mechanism to model global context information. However, pixels belonging to different classes usually have weak feature correlation. Modeling the global pixel-level correlation matrix indiscriminately is extremely redundant in the self-attention mechanism. In order to solve the above problem, we propose a hierarchical context network to differentially model homogeneous pixels with strong correlations and heterogeneous pixels with weak correlations. Specifically, we first propose a multi-scale guided pre-segmentation module to divide the entire feature map into different classed-based homogeneous regions. Within each homogeneous region, we design the pixel context module to capture pixel-level correlations. Subsequently, different from the self-attention mechanism that still models weak heterogeneous correlations in a dense pixel-level manner, the region context module is proposed to model sparse region-level dependencies using a unified representation of each region. Through aggregating fine-grained pixel context features and coarse-grained region context features, our proposed network can not only hierarchically model global context information but also harvest multi-granularity representations to more robustly identify multi-scale objects. We evaluate our approach on Cityscapes and the ISPRS Vaihingen dataset. Without Bells or Whistles, our approach realizes a mean IoU of 82.8% and overall accuracy of 91.4% on Cityscapes and ISPRS Vaihingen test set, achieving state-of-the-art results.

Digital twins of multiscale 3D heterogeneous geological objects: 3D simulations and seismic imaging of faults, fractures and caves

Numerical simulation of seismic waves’ propagation and imaging in a three-dimensional multiscale geological media is one of the main trends in the development of modern geophysics. We present the work flow for construction of the 3D digital twin of some real geological object. This object is cavernous fractured carbonate reservoir near the Riphean roof. One of the main features of this object is the presence of the family of faults filled with tectonic breccia. To simulate them, in particular, their interiors and surrounding damaged areas, we simulate paleotectonic processes with the use of 3D discrete elements technique. This simulation we do by GPU parallelization.Next, when the fine digital geological model is done, we perform 3D finite-difference simulation of the seismic waves’ propagation and get the full 3D synthetic data set. To do this simulation we use an original technique based on local grid refinement in time and space. To be able to deal with huge amount of input/output data we use High-Performance Computing (HPC) systems with parallel architecture and hybrid parallelization strategy by simultaneous use of MPI and OpenMP.The final step is validation of our original multiscale algorithm for seismic imaging, which suppresses specular reflection, but accumulates weak scattered/diffracted waves. This approach opens the way to the reconstruction of subseismic geological objects, like fractures, fracture corridors and clusters of caves. To construct these images we use HPC with parallel architecture and MPI+OpenMP programming.

Regional attention‐based single shot detector for SAR ship detection

Automatic ship detection in SAR imagery has been playing a significant role in the field of marine monitoring but great challenges still exist in real-time application. Despite the exciting progresses made by deep-learning techniques, most detectors failed to yield locations of fairly high quality. Moreover, the ships with variant sizes and aspects are easily omitted especially for small objects under complicated background. To alleviate the above problem, the authors propose an elaborately designed single shot detection framework combined with attention mechanism, which roughly locates the regions of interest via an automatically learned attentional map. This lay the foundation of accurate positioning of extremely small objects since the background interference can be effectively suppressed. Furthermore, a multi-level feature fusion module integrated in top-down and bottom-up manner is adopted to adequately aggregate features from not only adjacent but also distant layers. This strengthens local details and merge strong semantic information, enabling the generation of higher qualified anchors for the efficient detection of multi-scale and multi-orientated objects. Experiments on SAR ship dataset have achieved a promising result, surpassing current state-of-the-art methods.

Excavating effective information in different stage of backbone to improve semantic segmentation results

To overcome the two difficulties in semantic segmentation task:1) the presence of multi-scale objects, 2) the loss of spatial resolution, a new semantic segmentation model is designed in this paper, which can explore effective information from different stage of backbone. Extensive experiments conducted on two public benchmark datasets have proved the effectiveness of each module, and the new model achieves 73.62% mIoU on Cityscapes test set and 79.88% mIoU on PASCAL VOC 2012 test set.

A New SDIE Based on CFIE for Electromagnetic Scattering From IBC Objects

This article presents a new self-dual integral equation (SDIE) for electromagnetic scattering from arbitrarily impedance boundary condition (IBC) objects including partly coated objects. The proposed SDIE is constructed by using the combined field integral equation (CFIE) and IBC, shorted as C-SDIE. To overcome the difficulty of discontinuous surface impedance from nonuniform IBC/partly coated objects, the discontinuous Galerkin (DG) method is applied to discretize the C-SDIE. Numerical experiments confirm that the DG-C-SDIE has promising numerical performance in terms of accuracy and efficiency. Furthermore, the domain decomposition preconditioning based on DG is employed to further enhance the proposed DG-C-SDIE for large-scale, multi-scale objects. The numerical results demonstrate the capability of the proposed DG-C-SDIE.

Real-Time Water Surface Object Detection Based on Improved Faster R-CNN.

In this paper, we consider water surface object detection in natural scenes. Generally, background subtraction and image segmentation are the classical object detection methods. The former is highly susceptible to variable scenes, so its accuracy will be greatly reduced when detecting water surface objects due to the changing of the sunlight and waves. The latter is more sensitive to the selection of object features, which will lead to poor generalization as a result, so it cannot be applied widely. Consequently, methods based on deep learning have recently been proposed. The River Chief System has been implemented in China recently, and one of the important requirements is to detect and deal with the water surface floats in a timely fashion. In response to this case, we propose a real-time water surface object detection method in this paper which is based on the Faster R-CNN. The proposed network model includes two modules and integrates low-level features with high-level features to improve detection accuracy. Moreover, we propose to set the different scales and aspect ratios of anchors by analyzing the distribution of object scales in our dataset, so our method has good robustness and high detection accuracy for multi-scale objects in complex natural scenes. We utilized the proposed method to detect the floats on the water surface via a three-day video surveillance stream of the North Canal in Beijing, and validated its performance. The experiments show that the mean average precision (MAP) of the proposed method was 83.7%, and the detection speed was 13 frames per second. Therefore, our method can be applied in complex natural scenes and mostly meets the requirements of accuracy and speed of water surface object detection online.

A Discontinuous Galerkin Self-Dual Integral Equation Method for Scattering From IBC Objects

A discontinuous Galerkin self-dual integral equation (DG-SDIE) method is presented for accurately calculating electromagnetic scattering from objects with impedance boundary condition (IBC) surfaces. The working mechanisms of each term in the DG-SDIE formulation are studied physically and numerically. Based on the mechanisms, we propose a new efficient nonsymmetric DG-SDIE formulation (NDG-SDIE) for IBC problems. A comprehensive study is made to compare the NDG-SDIE with the previous symmetric DG-SDIE (SDG-SDIE) and antisymmetric DG-SDIE (ADG-SDIE) extension formulations for IBC. Numerical results demonstrate that NDG-SDIE is more efficient than SDG-SDIE and ADG-SDIE while generally maintaining the similar accuracy. In addition, we show that the DG methods are more attractive than the standard SDIE due to its flexible performance on large complex multiscale objects.

A Multilayer Fusion Light-Head Detector for SAR Ship Detection.

Synthetic aperture radar (SAR) ship detection is a heated and challenging problem. Traditional methods are based on hand-crafted feature extraction or limited shallow-learning features representation. Recently, with the excellent ability of feature representation, deep neural networks such as faster region based convolution neural network (FRCN) have shown great performance in object detection tasks. However, several challenges limit the applications of FRCN in SAR ship detection: (1) FRCN with a fixed receptive field cannot match the scale variability of multiscale SAR ship objects, and the performance degrade when the objects are small; (2) as a two-stage detector, FRCN performs an intensive computation and leads to low-speed detection; (3) when the background is complex, the imbalance of easy and hard examples will lead to a high false detection. To tackle the above issues, we design a multilayer fusion light-head detector (MFLHD) for SAR ship detection. Instead of using a single feature map, shallow high-resolution and deep semantic feature are combined to produce region proposal. In detection subnetwork, we propose a light-head detector with large-kernel separable convolution and position sensitive pooling to improve the detection speed. In addition, we adapt focal loss to loss function and training more hard examples to reduce the false alarm. Extensive experiments on SAR ship detection dataset (SSDD) show that the proposed method achieves superior performance in SAR ship detection both in accuracy and speed.

An Efficient Preconditioner Based on Adaptive Grouping Technique for Multiscale Problems

It is well known that the matrix condition number of discretized formulation for analyzing multiscale objects is very large, because the multiscale objects include the interaction between discrete unknowns in different sizes. The traditional efficient preconditioners for analysis of homogeneous objects are not efficient anymore or cannot be directly extended for the solution of multiscale objects. A good quality compressed block decomposition preconditioner is constructed to speed up the convergence rate of Krylov iterations in this communication by using the adaptive grouping technique. Meanwhile, the multilevel mymargin QR algorithm is utilized to accelerate the matrix-vector product operations. A series of numerical results show that this novel preconditioner is very effective in electromagnetic scattering calculation and can reduce simulation time significantly.

Class-Specific Anchor Based and Context-Guided Multi-Class Object Detection in High Resolution Remote Sensing Imagery with a Convolutional Neural Network

In this paper, the problem of multi-scale geospatial object detection in High Resolution Remote Sensing Images (HRRSI) is tackled. The different flight heights, shooting angles and sizes of geographic objects in the HRRSI lead to large scale variance in geographic objects. The inappropriate anchor size to propose the objects and the indiscriminative ability of features for describing the objects are the main causes of missing detection and false detection in multi-scale geographic object detection. To address these challenges, we propose a class-specific anchor based and context-guided multi-class object detection method with a convolutional neural network (CNN), which can be divided into two parts: a class-specific anchor based region proposal network (RPN) and a discriminative feature with a context information classification network. A class-specific anchor block providing better initial values for RPN is proposed to generate the anchor of the most suitable scale for each category in order to increase the recall ratio. Meanwhile, we proposed to incorporate the context information into the original convolutional feature to improve the discriminative ability of the features and increase classification accuracy. Considering the quality of samples for classification, the soft filter is proposed to select effective boxes to improve the diversity of the samples for the classifier and avoid missing or false detection to some extent. We also introduced the focal loss in order to improve the classifier in classifying the hard samples. The proposed method is tested on a benchmark dataset of ten classes to prove the superiority. The proposed method outperforms some state-of-the-art methods with a mean average precision (mAP) of 90.4% and better detects the multi-scale objects, especially when objects show a minor shape change.

Enet-CRF-Lidar: Lidar and Camera Fusion for Multi-Scale Object Recognition

Object recognition, which mainly includes object detection and semantic segmentation, is one of the critical challenges for intelligent vehicles. In most cases, cameras and Lidar are the most common sensors used for object recognition. However, both cameras and Lidar suffer from some inherent drawbacks. Therefore, the fusion of camera and Lidar becomes a natural solution to overcome the inherent defects of each single sensing modality. With the boost of deep learning-based algorithms, multi-sensor fusion methodologies employ deep-learning methods as their fusion strategy, which has made impressive accomplishments on large-scale objects such as vehicles and buses. However, most existing sensor-fusion strategies have the problem of ignoring detailed information caused by down-sampling operations in deep learning, which results in poor detection performance on small-scale objects such as pedestrians and cyclists. In this paper, we propose a real-time multi-sensor (Lidar and color camera) fusion strategy for multi-scale object recognition at the semantic level named Enet-CRF-Lidar. Firstly, a multi-module Enet is designed to adapt both large-scale objects and small-scale ones. Then, the CRF-RNN module is integrated with the multi-module Enet to introduce the low-level details of the input data, which leads to a significant improvement in small-scale object recognition. The experimental results show that the proposed Enet-CRF-Lidar module can provide reliable detection performance on multi-scale objects and can be adapted to complex scenarios.

Towards Accurate High Resolution Satellite Image Semantic Segmentation

Satellite image semantic segmentation, including extracting road, detecting building, and identifying land cover types, is essential for sustainable development, agriculture, forestry, urban planning, and climate change research. Nevertheless, it is still unclear how to develop a refined semantic segmentation model in an efficient and elegant way. In this paper, we propose attention dilation-LinkNet (AD-LinkNet) neural network that adopts encoder–decoder structure, serial–parallel combination dilated convolution, channel-wise attention mechanism, and pretrained encoder for semantic segmentation. Serial–parallel combination dilated convolution enlarges receptive field as well as assemble multi-scale features for multi-scale objects, such as long-span road and small pool. The channel-wise attention mechanism is designed to advantage the context information in the satellite image. The experimental results on road extraction and surface classification data sets prove that the AD-LinkNet shows a significant effect on improving the segmentation accuracy. We defeated the D-Linknet algorithm that won the first place in the CVPR 2018 DeepGlobe road extraction competition.

A Novel Hybrid Method of Spatially Filtered FDTD and Subgridding Technique

A novel hybrid method of spatially filtered finite-difference time-domain (SF-FDTD) scheme and subgridding technique is presented to analyze the multiscale objects. Because of the controllable stability of the SF-FDTD scheme, which is applied in the subgrid region, the uniform temporal increment can be employed in the whole simulation region. Therefore, the temporal interpolation of the field components can be avoided at the coarse-dense grids connecting boundary. In addition, by selecting the appropriate spatial interpolation field points in the Yee cells and only eliminating the unstable frequency domain magnetic fields, the influence of the filtering operation at the connecting boundary is removed. The simulation results are presented to validate the accuracy and efficiency of the hybrid method.

Rectangular-Normalized Superpixel Entropy Index for Image Quality Assessment.

Image quality assessment (IQA) is a fundamental problem in image processing that aims to measure the objective quality of a distorted image. Traditional full-reference (FR) IQA methods use fixed-size sliding windows to obtain structure information but ignore the variable spatial configuration information. In order to better measure the multi-scale objects, we propose a novel IQA method, named RSEI, based on the perspective of the variable receptive field and information entropy. First, we find that consistence relationship exists between the information fidelity and human visual of individuals. Thus, we reproduce the human visual system (HVS) to semantically divide the image into multiple patches via rectangular-normalized superpixel segmentation. Then the weights of each image patches are adaptively calculated via their information volume. We verify the effectiveness of RSEI by applying it to data from the TID2008 database and denoise algorithms. Experiments show that RSEI outperforms some state-of-the-art IQA algorithms, including visual information fidelity (VIF) and weighted average deep image quality measure (WaDIQaM).

Fast object detection based on binary deep convolution neural networks

In this study, a fast object detection algorithm based on binary deep convolution neural networks (CNNs) is proposed. Convolution kernels of different sizes are used to predict classes and bounding boxes of multi-scale objects directly in the last feature map of a deep CNN. In this way, rapid object detection with acceptable precision loss is achieved. In addition, binary quantisation for weight values and input data of each layer is used to squeeze the networks for faster object detection. Compared to full-precision convolution, the proposed binary deep CNNs for object detection results in 62 times faster convolutional operations and 32 times memory saving in theory, what's more, the proposed method is easy to be implemented in embedded computing systems because of the binary operation for convolution and low memory requirement. Experimental results on Pascal VOC2007 validate the effectiveness of the authors’ proposed method.

Multi-Scale Object Histogram Distance for LCCD Using Bi-Temporal Very-High-Resolution Remote Sensing Images

To improve the performance of land-cover change detection (LCCD) using remote sensing images, this study utilises spatial information in an adaptive and multi-scale manner. It proposes a novel multi-scale object histogram distance (MOHD) to measure the change magnitude between bi-temporal remote sensing images. Three major steps are related to the proposed MOHD. Firstly, multi-scale objects for the post-event image are extracted through a widely used algorithm called the fractional net evaluation approach. The pixels within a segmental object are taken to construct the pairwise frequency distribution histograms. An arithmetic frequency-mean feature is then defined from the red, green and blue band histogram. Secondly, bin-to-bin distance is adapted to measure the change magnitude between the pairwise objects of bi-temporal images. The change magnitude image (CMI) of the bi-temporal images can be generated through object-by-object. Finally, the classical binary method Otsu is used to divide the CMI to a binary change detection map. Experimental results based on two real datasets with different land-cover change scenes demonstrate the effectiveness of the proposed MOHD approach in detecting land-cover change compared with three widely used existing approaches.

Cloud and cloud shadow detection using multilevel feature fused segmentation network

Cloud and cloud shadow detection in remote sensing imagery is important for its wide range of applications. Traditionally, the detection is usually based on the manually designed thresholds from multiband, which is complicated and of multistage. To simplify the process of cloud and cloud shadow detection and improve the performance, we propose a multilevel feature fused segmentation network (MFFSNet), which can be trained end-to-end without any hand-tuned parameters. Specifically, a fully convolutional network is proposed for cloud and cloud shadow features learning. Then, we utilize a novel pyramid pooling module to extract contextual relation between cloud and shadow. Furthermore, a special multilevel feature fused structure is designed to combine semantic information with spatial information from different levels, so that we can better handle the multiscale objects and produce detailed segmentation boundaries. Experiments show that the MFFSNet outperforms the state-of-the-art methods and achieves high accuracies of 98.69% and 98.92% for cloud and cloud shadow detection.

Concise feature pyramid region proposal network for multi-scale object detection

Object detection is a hot research issue in the field of computer vision. Many methods focus on detecting large objects. And features of small objects are easily weakened or even disappeared after multiple convolution layers. So the detection rate of multi-scale objects is unsatisfied. Aiming at this problem, a concise feature pyramid region proposal network (CFPRPN) is proposed to address the problem of small objects detection in this paper without missing the large objects. In the process of object detection, we propose a new method of adjustment for the object location. So the balanced detection of multi-scale objects is realized. CFPRPN combines image pyramids and feature pyramids. An image pyramid consists of scaled versions of an image and the feature pyramids produce multiple layers’ feature maps. They are both conducive to capturing the feature information of small objects in deep convolutional networks. At the same time, proposals of overlapping sizes from different layers are applied to improve the recall rate of multi-scale objects. These series operations are beneficial for CFPRPN to extract better proposals. We experimentally prove that after adding the fine-tuning location, the detection rate of multi-scale object is further improved. The inspiring thing is that refining location method is suitable for most algorithms of object detection.

Stepwise Evolution Analysis of the Region-Merging Segmentation for Scale Parameterization

Multiscale image segmentation is a fundamental process of object-based image analysis, and the segmentation scale is a key parameter. The selection of the optimal scale parameter has attracted widely attentions in recent years, and many methods were proposed for this task. However, these methods usually need to set a scale sequence, which depends on the segmentation algorithm. When using another segmentation algorithm, the definition of the scale is different, these methods are less effective. In this study, we are aiming at developing a general framework for analyzing and estimating the optimal scale parameter for the region-merging segmentation, which is the most widely used approach for generating multiscale objects in remote sensing. A stepwise evolution analysis (SEA) framework and its application in optimal scale parameter estimation are proposed. In the SEA framework, the evolutions of scale, local variance (LV), and Moran's I (MI) indexes are analyzed step-by-step, and then, be used for automated scale parameterizations. Four LV- and MI-metrics-based methods were technically integrated in this framework, and applied to a QuickBird and a WorldView-2 image to evaluate the performance. The results demonstrated that: The stepwise analysis of a hierarchical region merging provided the distribution information of the scale, which could be used to estimate the key parameters and integrate different scale parameterization methods. The traditional LV- and MI-based scale estimation methods could be technically integrated, which showed the powerful ability and great potential in utilizing different scale parameterization methods for the region-merging segmentation.

Geospatial Object Detection in High Resolution Satellite Images Based on Multi-Scale Convolutional Neural Network

Daily acquisition of large amounts of aerial and satellite images has facilitated subsequent automatic interpretations of these images. One such interpretation is object detection. Despite the great progress made in this domain, the detection of multi-scale objects, especially small objects in high resolution satellite (HRS) images, has not been adequately explored. As a result, the detection performance turns out to be poor. To address this problem, we first propose a unified multi-scale convolutional neural network (CNN) for geospatial object detection in HRS images. It consists of a multi-scale object proposal network and a multi-scale object detection network, both of which share a multi-scale base network. The base network can produce feature maps with different receptive fields to be responsible for objects with different scales. Then, we use the multi-scale object proposal network to generate high quality object proposals from the feature maps. Finally, we use these object proposals with the multi-scale object detection network to train a good object detector. Comprehensive evaluations on a publicly available remote sensing object detection dataset and comparisons with several state-of-the-art approaches demonstrate the effectiveness of the presented method. The proposed method achieves the best mean average precision (mAP) value of 89.6%, runs at 10 frames per second (FPS) on a GTX 1080Ti GPU.