Global Spatial Information Research Articles

Multi-layer and multi-order fine-grained feature learning for artwork attribute recognition

With the digital development of artwork, more and more artwork images are made available to the public. In iMet Collection Dataset which is part of the FGVC6 workshop at CVPR 2019, an artwork image is annotated by multiple labels. Because of the local and subtle differences between artwork images with different attribute labels, artwork attribute recognition can be considered as a fine-grained visual categorization (FGVC) task. In this paper, a multi-layer and multi-order network (MLMO-Net) is proposed to capture both first- and second-order information in the artwork images. First-order information can be used to characterize the global spatial information and second-order information can be used to characterize the local statistical information. Both of first- and second-order information from multiple layers are aggregated together in MLMO-Net. In our experiments, several convolutional neural network (CNN) architectures, such as Vgg16 and ResNet50, and recent FGVC methods, such as the bilinear CNN, hierarchical bilinear CNN, and Navigator–teacher–scrutinizer network (NTS-Net), are tested on the iMet Collection 2019 Dataset. Experimental results shown that MLMO-Net achieves improvements over baseline methods. Through the research of this paper, a direction to improve the performance of artwork attribute recognition could be provided.

Spectral and Spatial Global Context Attention for Hyperspectral Image Classification

Recently, hyperspectral image (HSI) classification has attracted increasing attention in the remote sensing field. Plenty of CNN-based methods with diverse attention mechanisms (AMs) have been proposed for HSI classification due to AMs being able to improve the quality of feature representations. However, some of the previous AMs squeeze global spatial or channel information directly by pooling operations to yield feature descriptors, which inadequately utilize global contextual information. Besides, some AMs cannot exploit the interactions among channels or positions with the aid of nonlinear transformation well. In this article, a spectral-spatial network with channel and position global context (GC) attention (SSGCA) is proposed to capture discriminative spectral and spatial features. Firstly, a spectral-spatial network is designed to extract spectral and spatial features. Secondly, two novel GC attentions are proposed to optimize the spectral and spatial features respectively for feature enhancement. The channel GC attention is used to capture channel dependencies to emphasize informative features while the position GC attention focuses on position dependencies. Both GC attentions aggregate global contextual features of positions or channels adequately, following a nonlinear transformation. Experimental results on several public HSI datasets demonstrate that the spectral-spatial network with GC attentions outperforms other related methods.

Spatial Rough Intuitionistic Fuzzy C-Means Clustering for MRI Segmentation

Medical image segmentation is the challenging problem in real-time applications due to the occurrence of noise and uncertainties between different tissues in the magnetic resonance images (MRI). To overcome the issue, spatial rough intuitionistic fuzzy C-means method has been proposed. The segmentation of MR brain image has been implemented by updating the MRI based on global spatial information of pixel with intuitionistic fuzzy c-means algorithm for segmenting cerebro spinal fluid, white matter and gray matter tissues. Intuitionistic fuzzy sets and rough sets have been used to deal with uncertainty and vagueness in medical images. Intuitionistic fuzzy sets are used for image representations by using non-membership value, hesitation along with the membership value for the MR image. The membership value and non-membership value have been obtained using fuzzy hexagonal membership and fuzzy complement function respectively. Further, roughness measures are done to determine the initial cluster centroids by considering lower and upper approximation and the fuzzy c-means clustering algorithm has been updates by the euclidean distance between the pixels based on global spatial information for segmenting MR brain image. The proposed method have been implemented and analysed with quantitative and qualitatively for the synthetic and real MR images. Experimental results exhibit a higher degree of segmentation accuracy on both synthetic and real MR images compared to existing methods and achieves better performance.

Multi-Temporal Built-Up Grids of Brazilian Cities: How Trends and Dynamic Modelling Could Help on Resilience Challenges?

The northeastern Brazilian region has been vulnerable to hydrometeorological extremes, especially droughts, for centuries. A combination of natural climate variability (most of the area is semi-arid) and water governance problems increases extreme events’ impacts, especially in urban areas. Spatial analysis and visualisation of possible land-use change (LUC) zones and trends (urban growth vectors) can be useful for planning actions or decision-making policies for sustainable development. The Global Human Settlement Layer (GHSL) produces global spatial information, evidence-based analytics, and knowledge describing Earth’s human presence. In this work, the GHSL built-up grids for selected Brazilian cities were used to generate urban models using GIS (geographic information system) technologies and cellular automata for spatial pattern simulations of urban growth. In this work, six Brazilian cities were selected to generate urban models using GIS technologies and cellular automata for spatial pattern simulations of urban sprawl. The main goal was to provide predictive scenarios for water management (including simulations) and urban planning in a region highly susceptible to extreme hazards, such as floods and droughts. The northeastern Brazilian cities’ analysis raises more significant challenges because of the lack of land-use change field data. Findings and conclusions show the potential of dynamic modelling to predict scenarios and support water sensitive urban planning, increasing cities’ coping capacity for extreme hazards.

Attentive generative adversarial network for removing thin cloud from a single remote sensing image

Land-surface observation is easily affected by the light transmission and scattering of semi-transparent clouds, high or low, resulting in blurring and reduced contrast of ground objects. To improve the visual appearance of remote sensing images, the authors present a deep learning method for thin cloud removal using a new attentive generative adversarial network without prior knowledge or assumptions, which copes with thin clouds that are unevenly distributed on different images and learns the attention map with weighted information about spatial features. Such a spatial attention model can endow each pixel with the global spatial context information. Consequently, the generative network focuses on the thin cloud regions to generate better local image restoration, and the discriminative network can evaluate the local consistency of the repaired regions. The experimental results show that this method is superior to state-of-the-art methods in recovering detailed texture information.

SA-JSTN: Self-Attention Joint Spatiotemporal Network for Temperature Forecasting

The rapid development of remote sensing technology has brought abundant data support for deep learning based temperature forecasting research. However, recently proposed methods usually focus on the temporal relationship among temperature observation information, whereas ignore the spatial positions of different regions. Motivated by the observation that adjacent regions usually present similar temperature trends, in this article, we consider the temperature forecasting as a spatiotemporal sequence prediction problem, and propose a new deep learning model for temperature forecasting, self-attention joint spatiotemporal network (SA-JSTN), which simultaneously captures the spatiotemporal interdependency information. The kernel component of the SA-JSTN is a newly developed spatiotemporal memory (STM) unit, which describes the temporal and spatial models via a unified memory cell. STM is constructed based on the units of the convolutional long short-term memory (ConvLSTM). Instead of using simple convolutions for spatial information extraction, in STM, we improve ConvLSTM by a self-attention module, which has significantly enhanced the global spatial information representation ability of our proposed network. Compared with other deep learning based temperature forecasting methods, SA-JSTN is able to integrate the global spatial correlation into the temperature series prediction problem, and thus present better performance especially in short-term prediction. We have conducted comparison experiments on two typical temperature datasets to validate the effectiveness of our proposed method.

DIKA-Nets: Domain-invariant knowledge-guided attention networks for brain skull stripping of early developing macaques

As non-human primates, macaques have a close phylogenetic relationship to human beings and have been proven to be a valuable and widely used animal model in human neuroscience research. Accurate skull stripping (aka. brain extraction) of brain magnetic resonance imaging (MRI) is a crucial prerequisite in neuroimaging analysis of macaques. Most of the current skull stripping methods can achieve satisfactory results for human brains, but when applied to macaque brains, especially during early brain development, the results are often unsatisfactory. In fact, the early dynamic, regionally-heterogeneous development of macaque brains, accompanied by poor and age-related contrast between different anatomical structures, poses significant challenges for accurate skull stripping. To overcome these challenges, we propose a fully-automated framework to effectively fuse the age-specific intensity information and domain-invariant prior knowledge as important guiding information for robust skull stripping of developing macaques from 0 to 36 months of age. Specifically, we generate Signed Distance Map (SDM) and Center of Gravity Distance Map (CGDM) based on the intermediate segmentation results as guidance. Instead of using local convolution, we fuse all information using the Dual Self-Attention Module (DSAM), which can capture global spatial and channel-dependent information of feature maps. To extensively evaluate the performance, we adopt two relatively-large challenging MRI datasets from rhesus macaques and cynomolgus macaques, respectively, with a total of 361 scans from two different scanners with different imaging protocols. We perform cross-validation by using one dataset for training and the other one for testing. Our method outperforms five popular brain extraction tools and three deep-learning-based methods on cross-source MRI datasets without any transfer learning.

Human action recognition based on 3D body mask and depth spatial-temporal maps

In this paper, a method based on depth spatial-temporal maps(DSTMs) is presented for human action recognition from depth video sequences, which provides compact global spatial and temporal information of human motion for action recognition. In our approach, the initial frame of depth sequences is dilated to generate 3D body mask. The new depth sequences of major part of the human body are then computed after using 3D body mask on each depth frame. We project each frame of the new depth sequences onto three orthogonal axes to get three binary lists. Under each projection axis, binary lists are stitching in order through an entire depth sequence forming a DSTM. We evaluate our method on two standard databases. Experimental results show that this method could effectively capture the spatial and temporal information of human motion and improve the accuracy of human action recognition.

Spectral-spatial classification method for hyperspectral images using stacked sparse autoencoder suitable in limited labelled samples situation

Recently, deep learning (DL)-based methods have attracted increasing attention for hyperspectral images (HSIs) classification. However, the complex structure and limited number of labelled training samples of HSIs negatively affect the performance of DL models. In this paper, a spectral-spatial classification method is proposed based on the combination of local and global spatial information, including extended multi-attribute profiles and multiscale Gabor features, with sparse stacked autoencoder (GEAE). GEAE stacks the spatial and spectral information to form the fused features. Also, GEAE generates virtual samples using weighted average of available samples for expanding the training set so that many parameters of DL network can be learned optimally in limited labelled samples situations. Therefore, the similarity between samples is determined with distance metric learning to overcome the problems of Euclidean distance-based similarity metrics. The experimental results on three HSIs datasets demonstrate the effectiveness of the GEAE in comparison to some existing classification methods.

FPGA: Fast Patch-Free Global Learning Framework for Fully End-to-End Hyperspectral Image Classification

Deep learning techniques have provided significant improvements in hyperspectral image (HSI) classification. The current deep learning based HSI classifiers follow a patch-based learning framework by dividing the image into overlapping patches. As such, these methods are local learning methods, which have a high computational cost. In this paper, a fast patch-free global learning (FPGA) framework is proposed for HSI classification. In FPGA, an encoder-decoder based FCN is utilized to consider the global spatial information by processing the whole image, which results in fast inference. However, it is difficult to directly utilize the encoder-decoder based FCN for HSI classification as it always fails to converge due to the insufficiently diverse gradients caused by the limited training samples. To solve the divergence problem and maintain the abilities of FCN of fast inference and global spatial information mining, a global stochastic stratified sampling strategy is first proposed by transforming all the training samples into a stochastic sequence of stratified samples. This strategy can obtain diverse gradients to guarantee the convergence of the FCN in the FPGA framework. For a better design of FCN architecture, FreeNet, which is a fully end-to-end network for HSI classification, is proposed to maximize the exploitation of the global spatial information and boost the performance via a spectral attention based encoder and a lightweight decoder. A lateral connection module is also designed to connect the encoder and decoder, fusing the spatial details in the encoder and the semantic features in the decoder. The experimental results obtained using three public benchmark datasets suggest that the FPGA framework is superior to the patch-based framework in both speed and accuracy for HSI classification. Code has been made available at: https://github.com/Z-Zheng/FreeNet.

Image Deblurring Using Multi-Stream Bottom-Top-Bottom Attention Network and Global Information-Based Fusion and Reconstruction Network.

Image deblurring has been a challenging ill-posed problem in computer vision. Gaussian blur is a common model for image and signal degradation. The deep learning-based deblurring methods have attracted much attention due to their advantages over the traditional methods relying on hand-designed features. However, the existing deep learning-based deblurring techniques still cannot perform well in restoring the fine details and reconstructing the sharp edges. To address this issue, we have designed an effective end-to-end deep learning-based non-blind image deblurring algorithm. In the proposed method, a multi-stream bottom-top-bottom attention network (MBANet) with the encoder-to-decoder structure is designed to integrate low-level cues and high-level semantic information, which can facilitate extracting image features more effectively and improve the computational efficiency of the network. Moreover, the MBANet adopts a coarse-to-fine multi-scale strategy to process the input images to improve image deblurring performance. Furthermore, the global information-based fusion and reconstruction network is proposed to fuse multi-scale output maps to improve the global spatial information and recurrently refine the output deblurred image. The experiments were done on the public GoPro dataset and the realistic and dynamic scenes (REDS) dataset to evaluate the effectiveness and robustness of the proposed method. The experimental results show that the proposed method generally outperforms some traditional deburring methods and deep learning-based state-of-the-art deblurring methods such as scale-recurrent network (SRN) and denoising prior driven deep neural network (DPDNN) in terms of such quantitative indexes as peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) and human vision.

Airport risk of importation and exportation of the COVID-19 pandemic

On March 11, 2020, the Director-General of the World Health Organization (WHO) characterized the spread of the coronavirus disease, COVID-19, as a pandemic on the basis of “alarming levels of spread and severity, and by the alarming levels of inaction.” Hence, it is urgent and imperative to evaluate the risk of COVID-19 for as many global locations as possible. This study calculates the relative risk of the importation and exportation of the COVID-19 virus. The study's most important contribution is the calculation of the overall relative risk of the importation and exportation of COVID-19 from every airport in local municipalities around the world, based on global spatial and mapping information. Three scenarios of air travel restriction are considered, and the change in the risk of importation and exportation of COVID-19 is calculated. The relative risk of importation and exportation of COVID-19 clearly shows that not only China, Europe, Middle East, and East Asia, but also the U.S., Australia, and countries in northeast Asia and Latin America are subject to risk. Further, a larger reduction in air travel through airports in a large part of the cumulative incidence area would lead to a gradual decrease in the risk flow. Importantly, the exportation risk of the disease from some airports in China, Iran, and European countries has a higher global spread than the importation risk during the pandemic stage. Therefore, every airport, or government with airports in their jurisdiction, should implement strict countermeasures. It is also indispensable for these countries to undertake countermeasures for COVID-19, such as home quarantine within each country and restricting infected or suspected individuals from flying on airplanes.

Cascaded hybrid residual U-Net for glioma segmentation

Glioma segmentation is critical for making surgical plans. Recently, the traditional glioma segmentation method is less competitive with two deep learning segmentation strategies: the patch-based method which focuses more on the local feature for each pixel, and the image-based method which fully leverages the global feature and captures the overall shape, size and other characteristics of the lesion in a neighborhood of a pixel. In this study, we investigate and integrate the advantages of 2-D and 3-D image-based architectures, and propose a new convolutional neural network called the Cascaded Hybrid Residual U-Net (CHR-U-Net) for MRI glioma segmentation. The CHR-U-Net exploits both the 2D local features as well as the 3D global spatial contextual information simultaneously. In the first-level of CHR-U-Net, the R-2D-U-Net combines the 2D-U-Net and the residual unit for quick lesion area detecting without any miss. To prevent from missing false-positive pixels, the output of R-2D-U-Net is resampled by using the hard-mining to collect more possible false-positive samples. In the second-level of CHR-U-Net, the axial, coronal, and sagittal 3D-U-Nets are trained to predict whether pixels belong to the area of glioma. The results of three 3D-U-Nets are fused to improve the accuracy and reduce false positives. The database of 2017 BRATS challenge were used in our experiments for the verification. The Dices and Sensitivities of Enhancing, Whole, and Core areas were calculated. The Dices are 0.73, 0.90, and 0.83 and the Sensitivities are 0.83, 0.90, and 0.82, respectively, for the axial, coronal, and sagittal 3D-U-Nets. Experimental results show that the proposed model significantly improves the performance of glioma segmentation.

Global-and-Local Context Network for Semantic Segmentation of Street View Images.

Semantic segmentation of street view images is an important step in scene understanding for autonomous vehicle systems. Recent works have made significant progress in pixel-level labeling using Fully Convolutional Network (FCN) framework and local multi-scale context information. Rich global context information is also essential in the segmentation process. However, a systematic way to utilize both global and local contextual information in a single network has not been fully investigated. In this paper, we propose a global-and-local network architecture (GLNet) which incorporates global spatial information and dense local multi-scale context information to model the relationship between objects in a scene, thus reducing segmentation errors. A channel attention module is designed to further refine the segmentation results using low-level features from the feature map. Experimental results demonstrate that our proposed GLNet achieves 80.8% test accuracy on the Cityscapes test dataset, comparing favorably with existing state-of-the-art methods.

CEREBRUM: a fast and fully-volumetric Convolutional Encoder-decodeR for weakly-supervised sEgmentation of BRain strUctures from out-of-the-scanner MRI.

Many functional and structural neuroimaging studies call for accurate morphometric segmentation of different brain structures starting from image intensity values of MRI scans. Current automatic (multi-) atlas-based segmentation strategies often lack accuracy on difficult-to-segment brain structures and, since these methods rely on atlas-to-scan alignment, they may take long processing times. Alternatively, recent methods deploying solutions based on Convolutional Neural Networks (CNNs) are enabling the direct analysis of out-of-the-scanner data. However, current CNN-based solutions partition the test volume into 2D or 3D patches, which are processed independently. This process entails a loss of global contextual information, thereby negatively impacting the segmentation accuracy. In this work, we design and test an optimised end-to-end CNN architecture that makes the exploitation of global spatial information computationally tractable, allowing to process a whole MRI volume at once. We adopt a weakly supervised learning strategy by exploiting a large dataset composed of 947 out-of-the-scanner (3 Tesla T1-weighted 1mm isotropic MP-RAGE 3D sequences) MR Images. The resulting model is able to produce accurate multi-structure segmentation results in only a few seconds. Different quantitative measures demonstrate an improved accuracy of our solution when compared to state-of-the-art techniques. Moreover, through a randomised survey involving expert neuroscientists, we show that subjective judgements favour our solution with respect to widely adopted atlas-based software.

Channel Attention Module With Multiscale Grid Average Pooling for Breast Cancer Segmentation in an Ultrasound Image.

Breast cancer accounts for the second-largest number of deaths in women around the world, and more than 8% of women will suffer from the disease in their lifetime. Mortality due to breast cancer can be reduced by its early and precise diagnosis. Many studies have investigated methods for segmentation, and computer-aided diagnosis based on deep learning techniques, in particular, has recently gained attention. However, recently proposed methods such as fully convolutional network (FCN), SegNet, and U-Net still need to be further improved to provide better semantic segmentation when diagnosing breast cancer by ultrasound imaging, because of their low performance. In this article, we propose a channel attention module with multiscale grid average pooling (MSGRAP) for the precise segmentation of breast cancer regions in ultrasound images. We demonstrate the effectiveness of the channel attention module with MSGRAP for semantic segmentation and develop a novel semantic segmentation network with the proposed attention module for the precise segmentation of breast cancer regions in ultrasound images. While a conventional convolutional operation cannot use global spatial information on input images and only use the small local information in a kernel of a convolution filter, the proposed attention module allows using both global and local spatial information. In addition, through ablation studies, we come up with a network architecture for precise breast cancer segmentation in an ultrasound image. The proposed network was constructed with an open-source breast cancer ultrasound image data set, and its performance was compared with those of other state-of-the-art deep-learning models for the segmentation of breast cancer. The experimental results showed that our network outperformed other segmentation methods, and the proposed channel attention module improved the performance of the network for breast cancer segmentation in ultrasound images.

Bag of Color Features for Color Constancy

In this paper, we propose a novel color constancy approach, called Bag of Color Features (BoCF), building upon Bag-of-Features pooling. The proposed method substantially reduces the number of parameters needed for illumination estimation. At the same time, the proposed method is consistent with the color constancy assumption stating that global spatial information is not relevant for illumination estimation and local information ( edges, etc.) is sufficient. Furthermore, BoCF is consistent with color constancy statistical approaches and can be interpreted as a learning-based generalization of many statistical approaches. To further improve the illumination estimation accuracy, we propose a novel attention mechanism for the BoCF model with two variants based on self-attention. BoCF approach and its variants achieve competitive, compared to the state of the art, results while requiring much fewer parameters on three benchmark datasets: ColorChecker RECommended, INTEL-TUT version 2, and NUS8.

RETRACTED ARTICLE: Rough fuzzy region based bounded support fuzzy C-means clustering for brain MR image segmentation

Precise brain tissue segmentation and analysis in the presence of intensity non-uniformity (INU) and noise is the challenging task due to intensity overlaps between data pixels within the image. The clustering method is commonly used to variety of applications for grouping similar data items. Specifically, the fuzzy C-means (FCM) clustering method is extensively used in many real world and research applications. In this paper, the rough fuzzy region based bounded support fuzzy C-means (RFRBSFCM) clustering method is proposed for brain MR image INU estimation and correction, and segmentation. The rough fuzzy regions are estimated based on similarity distance vector and it is determined from both local and global spatial information. In addition, the proposed algorithm incorporates bounded support vector for estimating weighted image. The objective function of proposed algorithm is minimized for segmenting different tissues in brain MR image. The RFRBSFCM algorithm is tested with recent FCM clustering techniques using simulated T1 and T2-weighted brain MR images from public BrainWeb dataset. The quantitative results confirm that the proposed algorithm achieves superior performance than other recent state-of-the-art methods.

An Intuitionistic Kernel-Based Fuzzy C-Means Clustering Algorithm With Local Information for Power Equipment Image Segmentation

With the transformation of the national energy and power sector, the steady advancement of intelligent power grid construction and the continuous improvement of the Ubiquitous Power Internet of Things technology framework, it has further requirements for realizing state comprehensive awareness and efficient processing of information data, and has been widely used in power equipment. The infrared image recognition technology, which has been widely used in thermal fault diagnosis of power equipment, also requires deeper research. For traditional Intuitionistic Fuzzy C-means (IFCM) algorithm for image segmentation is sensitive to the clustering center lead to low final clustering precision and detail, the time complexity and the high shortage. The paper puts forward a kind of applicable to power equipment of the infrared image segmentation based on space distribution information of Intuitionistic Fuzzy clustering algorithm. Non-target objects with high intensity and uneven image intensity in infrared image have strong interference to image segmentation. The proposed algorithm can effectively suppress the interference. Firstly, the gaussian model is introduced into the global spatial distribution information of power equipment to improve the IFCM. Secondly, the spatial operator optimization membership function of local spatial information is used to solve the problem of edge blurring and uneven image intensity. Through experiments on the data set containing 300 infrared images of power equipment, the relative regional error rate is about 10%, which is less affected by the change of fuzzy factor m. The effectiveness and applicability of this algorithm are verified, which is obviously better than other comparison algorithms.

Dynamic Global-Local Spatial-Temporal Network for Traffic Speed Prediction

Predicting traffic speed accurately is a very challenging task of the intelligent traffic system (ITS), due to the complex and dynamic spatial-temporal dependencies from both temporal and spatial aspects. There not only exits short-term local neighboring fluctuation and long-term global trend in temporal aspect, but also local and global correlations in spatial aspect. Most existing work focus on the local spatial-temporal dependencies, ignoring the global dynamic spatial-temporal corrections, which is comparably critical for traffic speed prediction. To address this problem, we propose a novel Dynamic Global-Local Spatial-Temporal Network(DGLSTNet) for traffic speed prediction, which consists of multiple spatial-temporal module considering the local and global information simultaneously from both temporal and spatial perspective. Each temporal module applies stacked dilated convolution block to exploit multi-scale local temporal information. Moreover, we empoly a global temporal attention block to capture global dependencies of temporal domain in an attention mechanism. In each spatial module, we not only learn the local but also focus on dynamic global spatial information learned by dymamic graph learning block. Combining the feature results from local and global perspective, the capability and expressiveness of traffic predicting model is improved. Experiment results on two real-world traffic datasets have demonstrated that our proposed model can effectively capture the comprehensive spatial-temporal dependencies and can achieve state-of-the-art prediction performance compared with the existing works.