Global Image Information Research Articles

A hybrid level set model for image segmentation.

Active contour models driven by local binary fitting energy can segment images with inhomogeneous intensity, while being prone to falling into a local minima. However, the segmentation result largely depends on the location of the initial contour. We propose an active contour model with global and local image information. The local information of the model is obtained by bilateral filters, which can also enhance the edge information while smoothing the image. The local fitting centers are calculated before the contour evolution, which can alleviate the iterative process and achieve fast image segmentation. The global information of the model is obtained by simplifying the C-V model, which can assist contour evolution, thereby increasing accuracy. Experimental results show that our algorithm is insensitive to the initial contour position, and has higher precision and speed.

Single image super-resolution via hybrid resolution NSST prediction

Convolutional neural networks (CNNs) have achieved great success in single image super-resolution (SR). However, most previous methods predict high-resolution (HR) images in the spatial domain, producing over-smoothed outputs while losing texture details. To address this problem, in this paper we propose to predict nonsubsampled shearlet transform (NSST) coefficients, which better represent the global topology information and local texture details of HR images. On the other hand, we propose a deep hybrid resolution network by a residual-in-residual style, which aggregates features of multiple resolutions so as to gather rich context information in compact representations. When evaluated on a newly released RealSR dataset and traditional simulated datasets, our method, namely hybrid resolution NSST prediction (HRNP), achieves more appealing results, w.r.t. PSNR and SSIM, than the state-of-the-art methods. Moreover, we find our HRNP is more capable of preserving complex edges and curves than other methods.

An Explainable 3D Residual Self-Attention Deep Neural Network for Joint Atrophy Localization and Alzheimer's Disease Diagnosis Using Structural MRI.

Computer-aided early diagnosis of Alzheimer's disease (AD) and its prodromal form mild cognitive impairment (MCI) based on structure Magnetic Resonance Imaging (sMRI) has provided a cost-effective and objective way for early prevention and treatment of disease progression, leading to improved patient care. In this work, we have proposed a novel computer-aided approach for early diagnosis of AD by introducing an explainable 3D Residual Attention Deep Neural Network (3D ResAttNet) for end-to-end learning from sMRI scans. Different from the existing approaches, the novelty of our approach is three-fold: 1) A Residual Self-Attention Deep Neural Network has been proposed to capture local, global and spatial information of MR images to improve diagnostic performance; 2) An explainable method using Gradient-based Localization Class Activation mapping (Grad-CAM) has been introduced to improve the interpretability of the proposed method; 3) This work has provided a full end-to-end learning solution for automated disease diagnosis. Our proposed 3D ResAttNet method has been evaluated on a large cohort of subjects from real datasets for two changeling classification tasks (i.e. Alzheimer's disease (AD) vs. Normal cohort (NC) and progressive MCI (pMCI) vs. stable MCI (sMCI)). The experimental results show that the proposed approach has a competitive advantage over the state-of-the-art models in terms of accuracy performance and generalizability. The explainable mechanism in our approach is able to identify and highlight the contribution of the important brain parts (e.g., hippocampus, lateral ventricle and most parts of the cortex) for transparent decisions.

Mixed Noise Removal by Bilateral Weighted Sparse Representation

Image denoising is a fundamental but difficult task in image processing. Recovering a clean version from an image corrupted by mixed Gaussian and impulse noise is still challenging. The weighted sparse representation (WSR) method has been applied to deal with mixed noise and has achieved good performance. However, the WSR model presents an obvious disadvantage due to oversmoothing of the texture regions. In this paper, the novel bilateral weighted sparse representation model is presented for mixed noise removal. By introducing an image gradient-based weight, an adaptive sparse ratio is provided for each image patch to protect additional image details. In addition, a weighted method noise regularization term is proposed to utilize the global image information submerged in the method noise, which can improve the visual effect of recovered noise-free images. Both the subjective and objective performance are evaluated, and the experimental results show that the proposed method outperforms existing mixed noise removal algorithms, especially in terms of the visual performance.

PMAN: Progressive Multi-Attention Network for Human Pose Transfer

This paper presents a novel approach for human pose transfer, progressive multi-attention network (PMAN), which generates a new human image by transferring the pose of a given person to a target pose. The network gradually updates the pose feature and the image feature through a series of multi-attention transfer blocks (MATBs). Each MATB consists of two attention mechanisms: pose-conditioned batch normalization (PCBN) and cooperative attention mechanism (CAM). Specifically, in low-level feature space, the PCBN layer with pose information is used to replace the BN layer of the image channel to realize the preliminary guidance of pose to image. The CAM is implemented as two gated mechanisms in high-level feature space, which reveals the essence of human pose transfer, that is, mutual guidance and dynamic control between pose and image. Gated memory writing is used to calculate the pixel-wise weight of the pose by using global image information to guide the update of the pose. Gated response utilizes an adaptive gating mechanism to dynamically control the pose information flow so as to update the image. A large number of subjective and objective experiments on DeepFashion and Market-1501 demonstrate the superiority of our method. The proposed multi-attention mechanism is well adapted to the human pose transfer task and provides a possible new idea for other cross-domain generation tasks.

Identification of invisible ischemic stroke in noncontrast CT based on novel two‐stage convolutional neural network model

Early identification of ischemic stroke lesion regions plays a vital role in its treatments like thrombolytic therapy and patients' recovery. Noncontrast computed tomography (ncCT) is the most widespread imaging modality in emergency departments. Unfortunately, it is extremely hard to distinguish the lesion from healthy tissue during the hyper-acute phase of stroke. In this paper, a two-stage convolutional neural network-based method was proposed to identify the invisible ischemic stroke from ncCT. In order to combine the global and local information of images effectively, a cascaded structure with two coordinated networks was used to detect the suspicious stroke regions on the whole and optimize the detailed localization. In the first stage, an end-to-end U-net with adaptive threshold was proposed to integrate global position, symmetry and gray texture information to detect the suspicious regions. After reducing the interference from most normal regions, a ResNet-based patch classification network was used to eliminate some false positive samples on suspicious regions by mining deeper image features, contributing to a more precise localization of stroke. Finally, a MAP model was used to optimize the result by combining the classification results of each patch with their spatial constraint information. Three independent experiments, that is, training and testing on dataset from one hospital, on the combination of two, and on the two respectively, were performed on a total of 277 cases from two hospitals to validate the proposed model, The proposed method achieved identification accuracy of 91.89%, 87.21%, and 85.71% in the three experiments, and the final localization accuracy in terms of precise localization of stroke were 82.35%, 83.02%, and 81.40%, respectively, which indicated the robustness and clinical values of the method. There are some deep image feature differences between stroke region and normal region on ncCT images. The proposed two-stage convolutional neural network model can well seize these features and use them to effectively identify and locate stroke.

Landscape Similarity Analysis Using Texture Encoded Deep-Learning Features on Unclassified Remote Sensing Imagery

Convolutional neural networks (CNNs) are known for their ability to learn shape and texture descriptors useful for object detection, pattern recognition, and classification problems. Deeper layer filters of CNN generally learn global image information vital for whole-scene or object discrimination. In landscape pattern comparison, however, dense localized information encoded in shallow layers can contain discriminative information for characterizing changes across image local regions but are often lost in the deeper and non-spatial fully connected layers. Such localized features hold potential for identifying, as well as characterizing, process–pattern change across space and time. In this paper, we propose a simple yet effective texture-based CNN (Tex-CNN) via a feature concatenation framework which results in capturing and learning texture descriptors. The traditional CNN architecture was adopted as a baseline for assessing the performance of Tex-CNN. We utilized 75% and 25% of the image data for model training and validation, respectively. To test the models’ generalization, we used a separate set of imagery from the Aerial Imagery Dataset (AID) and Sentinel-2 for model development and independent validation. The classical CNN and the Tex-CNN classification accuracies in the AID were 91.67% and 96.33%, respectively. Tex-CNN accuracy was either on par with or outcompeted state-of-the-art methods. Independent validation on Sentinel-2 data had good performance for most scene types but had difficulty discriminating farm scenes, likely due to geometric generalization of discriminative features at the coarser scale. In both datasets, the Tex-CNN outperformed the classical CNN architecture. Using the Tex-CNN, gradient-based spatial attention maps (feature maps) which contain discriminative pattern information are extracted and subsequently employed for mapping landscape similarity. To enhance the discriminative capacity of the feature maps, we further perform spatial filtering, using PCA and select eigen maps with the top eigen values. We show that CNN feature maps provide descriptors capable of characterizing and quantifying landscape (dis)similarity. Using the feature maps histogram of oriented gradient vectors and computing their Earth Movers Distances, our method effectively identified similar landscape types with over 60% of target-reference scene comparisons showing smaller Earth Movers Distance (EMD) (e.g., 0.01), while different landscape types tended to show large EMD (e.g., 0.05) in the benchmark AID. We hope this proposal will inspire further research into the use of CNN layer feature maps in landscape similarity assessment, as well as in change detection.

Structured Computational Modeling of Human Visual System for No-reference Image Quality Assessment

Objective image quality assessment (IQA) plays an important role in various visual communication systems, which can automatically and efficiently predict the perceived quality of images. The human eye is the ultimate evaluator for visual experience, thus the modeling of human visual system (HVS) is a core issue for objective IQA and visual experience optimization. The traditional model based on black box fitting has low interpretability and it is difficult to guide the experience optimization effectively, while the model based on physiological simulation is hard to integrate into practical visual communication services due to its high computational complexity. For bridging the gap between signal distortion and visual experience, in this paper, we propose a novel perceptual no-reference (NR) IQA algorithm based on structural computational modeling of HVS. According to the mechanism of the human brain, we divide the visual signal processing into a low-level visual layer, a middle-level visual layer and a high-level visual layer, which conduct pixel information processing, primitive information processing and global image information processing, respectively. The natural scene statistics (NSS) based features, deep features and free-energy based features are extracted from these three layers. The support vector regression (SVR) is employed to aggregate features to the final quality prediction. Extensive experimental comparisons on three widely used benchmark IQA databases (LIVE, CSIQ and TID2013) demonstrate that our proposed metric is highly competitive with or outperforms the state-of-the-art NR IQA measures.

Random Region Matting for the High-Resolution PolSAR Image Semantic Segmentation

Polarimetric synthetic aperture radar (PolSAR) imagery can provide more intuitive and detailed SAR polarization information, and it is widely used in the classification and semantic segmentation of remote sensing. To bridge the PolSAR data and application, the 2020 Gaofen Challenge on Automated High-Resolution Earth Observation Image Interpretation provides a set of high-quality PolSAR semantic segmentation dataset. A series of preprocessing methods is first used to analyze the PolSAR images to improve the semantic segmentation performance of the PolSAR imagery. A special polarimetric decomposition method is used to extract the features, and the filter and the data truncation are implemented to enhance local and global information of images. And the random region matting method is proposed to expand the training samples. Finally, the DeepLabV3+ method with the ResNet101-V2 is employed to achieve the semantic segmentation. A variety of comparison experiments verifies the effectiveness of our methods. Simultaneously, compared with the classification methods of other groups in the competition, our methods have obvious advantages in the inference time and semantic segmentation accuracy. The proposed method achieved a frequency weighted intersection over union of 75.29% in the contest.

Infrared and Visible Image Fusion Method Based on ResNet in a Nonsubsampled Contourlet Transform Domain

Although the traditional image fusion method can obtain rich image results, obvious artificial noise and artifacts are often present in the resulting image. Fusion algorithms based on neural networks can avoid the shortcomings of traditional methods, but they are more complex and less flexible. In this study, we proposed a fusion method using the deep residual neural network ResNet152, which can not only effectively suppress artificial noise but also preserve the edge details of the image and improve the efficiency of the neural network. The proposed method is characterized by a multiscale transformation of an infrared image and visible light image in the optimized nonsubsampled contourlet transformation domain, and the deep residual neural network ResNet152 is used to extract the deep features of the low-pass component to guide the fusion of the low-pass component. The bandpass component is fused by taking the modulus maximum. This method can fully retain the global features and structural information of the source image in the result image. Compared to existing fusion methods using public test image sets, the experimental results show that on a subjective level, the fusion method creates sharper depth edges and fewer noise artifacts than traditional fusion methods. From an objective perspective, the average value for the results of the evaluation function is greater than that of other fusion methods.

Adaptive dynamic range shift (ADRIFT) quantitative phase imaging

Quantitative phase imaging (QPI) with its high-contrast images of optical phase delay (OPD) maps is often used for label-free single-cell analysis. Contrary to other imaging methods, sensitivity improvement has not been intensively explored because conventional QPI is sensitive enough to observe the surface roughness of a substrate that restricts the minimum measurable OPD. However, emerging QPI techniques that utilize, for example, differential image analysis of consecutive temporal frames, such as mid-infrared photothermal QPI, mitigate the minimum OPD limit by decoupling the static OPD contribution and allow measurement of much smaller OPDs. Here, we propose and demonstrate supersensitive QPI with an expanded dynamic range. It is enabled by adaptive dynamic range shift through a combination of wavefront shaping and dark-field QPI techniques. As a proof-of-concept demonstration, we show dynamic range expansion (sensitivity improvement) of QPI by a factor of 6.6 and its utility in improving the sensitivity of mid-infrared photothermal QPI. This technique can also be applied for wide-field scattering imaging of dynamically changing nanoscale objects inside and outside a biological cell without losing global cellular morphological image information.

Image Super-Resolution Using Lightweight Multiscale Residual Dense Network

The current super-resolution methods cannot fully exploit the global and local information of the original low-resolution image, resulting in loss of some information. In order to solve the problem, we propose a multiscale residual dense network (MRDN) for image super-resolution. This network is constructed based on the residual dense network. It can integrate the multiscale information of the image and avoid losing too much information in the deep level of the network, while extracting more information under different receptive fields. In addition, in order to reduce the redundancy of the network parameters of MRDN, we further develop a lightweight parameter method and deploy it at different scales. This method can not only reduce the redundancy of network parameters but also enhance the nonlinear mapping ability of the network at different scales. Thus, it can better learn and fit the feature information of the original image and recover the satisfactory super-resolution image. Extensive experiments are conducted, which demonstrate the effectiveness of the proposed method.

The synergy of double attention: Combine sentence-level and word-level attention for image captioning

The existing attention models of image captioning typically extract only word-level attention information, i.e., the attention mechanism extracts local attention information from the image to generate the current word, and lacks accurate image global information guidance. In this paper, we first propose an image captioning approach based on self-attention. Sentence-level attention information is extracted from the image through self-attention mechanism to represent the global image information needed to generate sentences. Furthermore, we propose a double attention model which combines sentence-level attention model with word-level attention model to generate more accurate captions. We implement supervision and optimization in the intermediate stage of the model to solve information interference problems. In addition, we perform two-stage training with reinforcement learning to optimize the evaluation metric of the model. Finally, we evaluated our model on three standard datasets, i.e., Flickr8k, Flickr30k and MSCOCO. Experimental results show that our double attention model can generate more accurate and richer captions, and outperforms many state-of-the-art image captioning approaches in various evaluation metrics.

Multi-scale deep relational reasoning for facial kinship verification

In this paper, we propose a deep relational network which exploits multi-scale information of facial images for kinship verification. Unlike most existing deep learning based facial kinship verification methods which employ convolutional neural networks to extract holistic features, we present a deep model to exploit facial kinship relationship from local regions. For each given pair of face images, our method uses two convolutional neural networks which share parameters to extract different scales of features, which are expected to provide global contextual information of face images. We split a set of features at the same scale into multiple groups, where different groups capture information of different local regions. For each pair of local feature groups which are extracted from the same scale and position, we propose a relation network to reason their relationship, and use a verification network to infer the kin relation based on the results of local relations from different facial regions. We conduct experiments on two widely used facial kinship datasets: KinFaceW-I and KinFaceW-II, and our experimental results are presented to demonstrate the effectiveness of our approach.

An integrated two-stage approach for image segmentation via active contours

A novel integrated two-stage approach is proposed for image segmentation, where the edge, global and local region information of images are in turn incorporated to define the intensity fitting energy. In the first stage, the Chan-Vese model flexibly assimilates the edge indicator function in the beginning, and then the Laplace operator is introduced to regularize the level set function when minimizing the energy functional. As an edge-based and global region-based active contour, it can be inclined to rapidly produce a coarse segmentation result. In the second stage, we further segment the image by absorbing the local region fitting energy, where its initialization is acquired by the final active contour of the first stage. In addition, we present a generalized level set regularization term, which efficiently eliminates the periodically re-initialization procedure of traditional level set methods and maintains the corresponding signed distance property. Compared with the first stage, the local object details are accurately segmented in the second stage, which can acquire an accurate segmentation result. Qualitative and quantitative experimental results demonstrate the accuracy, robustness and efficiency of our approach with applications to some synthetical and real-world images.

Thermal infrared colorization via conditional generative adversarial network

Transforming a thermal infrared image into a realistic RGB image is a challenging task. In this paper we propose a deep learning method to bridge this gap. We propose learning the transformation mapping using a coarse-to-fine generator that preserves the details. Since the standard mean squared loss cannot penalize the distance between colorized and ground truth images well, we propose a composite loss function that combines content, adversarial, perceptual and total variation losses. The content loss is used to recover global image information while the latter three losses are used to synthesize local realistic textures. Quantitative and qualitative experiments demonstrate that our approach significantly outperforms existing approaches on the KAIST multispectral pedestrian dataset, achieving more plausible RGB images. Our code is available online22https://github.com/Kuangxd/TICCGAN..

Multi-View Based Multi-Model Learning for MCI Diagnosis.

Mild cognitive impairment (MCI) is the early stage of Alzheimer’s disease (AD). Automatic diagnosis of MCI by magnetic resonance imaging (MRI) images has been the focus of research in recent years. Furthermore, deep learning models based on 2D view and 3D view have been widely used in the diagnosis of MCI. The deep learning architecture can capture anatomical changes in the brain from MRI scans to extract the underlying features of brain disease. In this paper, we propose a multi-view based multi-model (MVMM) learning framework, which effectively combines the local information of 2D images with the global information of 3D images. First, we select some 2D slices from MRI images and extract the features representing 2D local information. Then, we combine them with the features representing 3D global information learned from 3D images to train the MVMM learning framework. We evaluate our model on the Alzheimer’s Disease Neuroimaging Initiative (ADNI) database. The experimental results show that our proposed model can effectively recognize MCI through MRI images (accuracy of 87.50% for MCI/HC and accuracy of 83.18% for MCI/AD).

FAOD-Net: A Fast AOD-Net for Dehazing Single Image

In this paper, we present an extremely computation-efficient model called FAOD-Net for dehazing single image. FAOD-Net is based on a streamlined architecture that uses depthwise separable convolutions to build lightweight deep neural networks. Moreover, the pyramid pooling module is added in FAOD-Net to aggregate the context information of different regions of the image, thereby improving the ability of the network model to obtain the global information of the foggy image. To get the best FAOD-Net, we use the RESIDE training set to train our proposed model. In addition, we have carried out extensive experiments on the RESIDE test set. We use full-reference and no-reference image quality evaluation indicators to measure the effect of dehazing. Experimental results show that the proposed algorithm has satisfactory results in terms of defogging quality and speed.

Gated Fusion Network for Degraded Image Super Resolution

Single image super resolution aims to enhance image quality with respect to spatial content, which is a fundamental task in computer vision. In this work, we address the task of single frame super resolution with the presence of image degradation, e.g., blur, haze, or rain streaks. Due to the limitations of frame capturing and formation processes, image degradation is inevitable, and the artifacts would be exacerbated by super resolution methods. To address this problem, we propose a dual-branch convolutional neural network to extract base features and recovered features separately. The base features contain local and global information of the input image. On the other hand, the recovered features focus on the degraded regions and are used to remove the degradation. Those features are then fused through a recursive gate module to obtain sharp features for super resolution. By decomposing the feature extraction step into two task-independent streams, the dual-branch model can facilitate the training process by avoiding learning the mixed degradation all-in-one and thus enhance the final high-resolution prediction results. We evaluate the proposed method in three degradation scenarios. Experiments on these scenarios demonstrate that the proposed method performs more efficiently and favorably against the state-of-the-art approaches on benchmark datasets.

A Novel Active Contour Model Guided by Global and Local Signed Energy-Based Pressure Force

Active contour models (ACMs) have been widely applied in the field of image segmentation. However, it is still very challenging to construct an efficient ACM to segment images with intensity inhomogeneity. In this paper, a novel ACM guided by global and local signed energy-based pressure force (GLSEPF) is proposed. First, by computing the energy difference between the inner and outer energies of the evolution curve, a global signed energy-based pressure force (GSEPF) is designed, which can improve the robustness to initial curves. Second, a local signed energy-based pressure force (LSEPF) is introduced by computing the pixel-by-pixel energy difference within local neighborhood region, which can handle images with intensity inhomogeneity and noise. Finally, the global image information and the local energy information are used for the global and local force propagation functions, respectively. The global and local variances are used to automatically balance the weights of the GSEPF and the LSEPF, which can solve the problem of setting parameters. Meanwhile, a regularization term and a penalty term are applied to avoid the re-initialization process during iterations and smooth the level set function. Experimental results on different types of images demonstrate that the proposed model is more robust than the popular region-based and mixed ACMs for segmenting images with intensity inhomogeneity and noise. The code is available at: https://github.com/HuaxiangLiu/GLSEPF/.