Structure-preserving Image Research Articles

A Generalized Framework for Edge-Preserving and Structure-Preserving Image Smoothing

Image smoothing is a fundamental procedure in applications of both computer vision and graphics. The required smoothing properties can be different or even contradictive among different tasks. Nevertheless, the inherent smoothing nature of one smoothing operator is usually fixed and thus cannot meet the various requirements of different applications. In this paper, a non-convex non-smooth optimization framework is proposed to achieve diverse smoothing natures where even contradictive smoothing behaviors can be achieved. To this end, we first introduce the truncated Huber penalty function which has seldom been used in image smoothing. A robust framework is then proposed. When combined with the strong flexibility of the truncated Huber penalty function, our framework is capable of a range of applications and can outperform the state-of-the-art approaches in several tasks. In addition, an efficient numerical solution is provided and its convergence is theoretically guaranteed even the optimization framework is non-convex and non-smooth. The effectiveness and superior performance of our approach are validated through comprehensive experimental results in a range of applications.

An adaptive method for image restoration based on high-order total variation and inverse gradient

The total variation (TV) regularization model for image restoration is widely utilized due to its edge preservation properties. Despite its advantages, the TV regularization can obtain spurious oscillations in flat regions of digital images and thus recent works advocate high-order TV regularization models. In this work, we propose an adaptive image restoration method based on a combination of first-order and second-order total variations regularization with an inverse-gradient-based adaptive parameter. The proposed model removes noise effectively and preserves image structures. Due to the adaptive parameter estimation based on the inverse gradient, it avoids the staircasing artifacts associated with TV regularization and its variant models. Experimental results indicate that the proposed method obtains better restorations in terms of visual quality as well as quantitatively. In particular, our proposed adaptive higher-order TV method obtained (19.3159, 0.7172, 0.90985, 0.79934, 0.99838) PSNR, SSIM, MS-SSIM, F-SIM, and P-SIM values compared to related models such as the TV-Bounded Hessian (18.9735, 0.6599, 0.8718, 0.73833, 0.99767), and TV-Laplacian (19.0345, 0.6719, 0.88198, 0.75405, 0.99789).

Path-Based Analysis for Structure-Preserving Image Filtering

Structure-preserving image filtering is an image smoothing technique that aims to preserve prominent structures while removing unwanted details in natural images. However, relevant studies mainly focus on small variances/fluctuations suppression and are vulnerable to separate pixels connected by some low-contrast edges or cluster pixels which exhibit strong differences between neighbors in highly textured region. Inspired by the fact that the human visual system significantly outperforms manually designed operators in extracting meaningful structures from natural scenes, we present an efficient structure-preserving filtering method which integrates similarity, proximity and continuation principles of human perception to accomplish high-contrast details (textures/noises) smoothing. Additionally, a Liebig’s law of minimum-based distance transform is presented to seamlessly incorporate the three properties for the description of the filter kernel. Experiments demonstrate that our distance transform keeps a clustering-like manner of separating different image pixels and grouping similar ones with the awareness of structure. When integrating this affinity measure into the bilateral-filter-like framework, our method can efficiently remove high-contrast textures/noises while preserving major structures.

Comparison of Five Spatio-Temporal Satellite Image Fusion Models over Landscapes with Various Spatial Heterogeneity and Temporal Variation

In recent years, many spatial and temporal satellite image fusion (STIF) methods have been developed to solve the problems of trade-off between spatial and temporal resolution of satellite sensors. This study, for the first time, conducted both scene-level and local-level comparison of five state-of-art STIF methods from four categories over landscapes with various spatial heterogeneity and temporal variation. The five STIF methods include the spatial and temporal adaptive reflectance fusion model (STARFM) and Fit-FC model from the weight function-based category, an unmixing-based data fusion (UBDF) method from the unmixing-based category, the one-pair learning method from the learning-based category, and the Flexible Spatiotemporal DAta Fusion (FSDAF) method from hybrid category. The relationship between the performances of the STIF methods and scene-level and local-level landscape heterogeneity index (LHI) and temporal variation index (TVI) were analyzed. Our results showed that (1) the FSDAF model was most robust regardless of variations in LHI and TVI at both scene level and local level, while it was less computationally efficient than the other models except for one-pair learning; (2) Fit-FC had the highest computing efficiency. It was accurate in predicting reflectance but less accurate than FSDAF and one-pair learning in capturing image structures; (3) One-pair learning had advantages in prediction of large-area land cover change with the capability of preserving image structures. However, it was the least computational efficient model; (4) STARFM was good at predicting phenological change, while it was not suitable for applications of land cover type change; (5) UBDF is not recommended for cases with strong temporal changes or abrupt changes. These findings could provide guidelines for users to select appropriate STIF method for their own applications.

Adaptive iterative reconstruction based on relative total variation for low-intensity computed tomography

Low-intensity projections (high-level noise) may degrade computed tomography (CT) image quality in some applications. CT image reconstruction from projections with low intensity has therefore attracted attention from researchers, who aim to suppress noise and preserve structures. Selecting a regularization parameter is a significant step for iterative reconstruction. To address this problem, we propose a reconstruction model that employs relative total variation (RTV)-regularized projections onto convex sets (POCS-RTV). This reconstruction model can effectively suppress noise and preserve image structures. In POCS-RTV, the L-curve method can adaptively determine the regularization parameter, producing the adaptive POCS-RTV (POCS-ARTV) reconstruction model. A nested iterative algorithm is applied to solving the POCS-RTV and POCS-ARTV models. Compared to reconstruction models based on ℓ1 and ℓ0 pseudo norm, POCS-ARTV can acquire adaptive structure preservation and noise reduction, and it performs the best in terms of image quality assessment (IQA) indexes and graphic comparisons. Compared to the reconstruction model based on parallel primal-dual splitting algorithm for total generalized variation and shearlet (PPDS-TGVS) regularization, POCS-ARTV reconstructs images that are visually closer to the reference image. Experiments on digital phantoms and real CT projections verified superiorities of POCS-ARTV, and the ability of the L-curve method to determine an acceptable regularization parameter.

A comprehensive survey on non-photorealistic rendering and benchmark developments for image abstraction and stylization

This survey presents a comprehensive study on non-photorealistic rendering (NPR). NPR technique renders 2D input image into abstracted and artistic stylized images. NPR mainly dwells on image processing, computer vision, and visualizing the graphics processing techniques. The survey highlights the evolution of IA–AR system and the subsequent classification of NPR techniques. The survey also has cognized the various works done on stroke-based rendering, color image analogy, region-based rendering, image filtering abstraction, and stylization. The inference drawn from the survey is the computer system using a stylus to fully automatic structure-preserving image abstraction and stylization got evolved from a traditional method of human interaction. From the survey carried out on the most significant papers from 1963 to 2017, we felt the need for setting up of benchmark guidelines, the data set with varies subjective matters, and quality assessment techniques with various statistical essences. From the survey information pertaining to benchmark image characteristics, properties and their constraints with the contextual feature in an image have been identified. Finally, survey work listed out the NPR application in various fields of image processing and highlighted empirical challenges and hampers in NPR domain. This survey work has empowered us to proceed in the right direction and enthusiasm to bring forth the problem statement and carry out research work.

Nonconvex Low Rank Approximation With Phase Congruency Regularization for Mixed Noise Removal

Mixed noise removal from a natural image is a challenging task since the complex noise distribution usually is inestimable. Many noise removal methods based on the low rank approximation have an excellent image denoising performance and are effective for recovering the images corrupted by Gaussian noise. These methods based on the additive white Gaussian noise(AWGN) model are sensitive to the outliers and non-Gaussian noise, such as the salt-and-pepper impulse noise (SPIN) and random valued impulse noise (RVIN). Such methods for mixed noise removal, however, are less effective in preserving image structures and tend to undesired staircase artifacts. This paper presents a novel Nonconvex Low Rank Model with Phase congruency and overlapping Group sparsity regularization (NLRM-PG) for mixed noise removal. Moreover, an efficient optimization algorithm under the alternating direction method of multipliers and majorization minimization framework is proposed to solve the NLRM-PG model. We demonstrate that the proposed method is effective for preserving local irregular structures and it reduces staircase artifacts with the two types of mixture noise, namely, AWGN+SPIN and AWGN+RVIN. Both qualitative and quantitative experiment results on synthetic noisy images and real noisy images illustrate that the proposed method can remove mixture noise in images more efficiently than the existing methods can do. And the results also outperform those obtained by using the competing state-of-the-art methods, particularly for the removal of high-density impulse noise.

Learning a Deep Representative Saliency Map With Sparse Tensors

The past few years have witnessed the prosperity of establishing deep architectures for modeling complex structured data. In this paper, motivated by the hierarchical, multi-scale and sparse characteristics of Human Visual System (HVS), we advance a new Deep and Sparse Representative (DSR) saliency detection approach for natural images. Under the assumption that salient regions are considered as sparse representatives of an image, we gradually select the salient regions via a deep and hierarchical sparse coding model. Moreover, in order to well preserve image structures and achieve more efficient coding, we formulate image patches as tensors and propose a Sparse Tensor Coding based DSR (STC-DSR) approach to find representatives. A generalized Laplacian regularizer is further cast on the coding to refine the coefficients. Finally, the saliency map is formulated by a spatial aggregation and multi-scale fusion of multilayer representatives. The proposed STC-DSR is investigated on the synthetic and public datasets and the experimental results prove its efficiency and superiority to its counterparts.

Infrared Image Super-Resolution Reconstruction Based on Quaternion Fractional Order Total Variation with Lp Quasinorm

Owing to the limitations of the imaging principle as well as the properties of imaging systems, infrared images often have some drawbacks, including low resolution, a lack of detail, and indistinct edges. Therefore, it is essential to improve infrared image quality. Considering the information of neighbors, a description of sparse edges, and by avoiding staircase artifacts, a new super-resolution reconstruction (SRR) method is proposed for infrared images, which is based on fractional order total variation (FTV) with quaternion total variation and the L p quasinorm. Our proposed method improves the sparsity exploitation of FTV, and efficiently preserves image structures. Furthermore, we adopt the plug-and-play alternating direction method of multipliers (ADMM) and the fast Fourier transform (FFT) theory for the proposed method to improve the efficiency and robustness of our algorithm; in addition, an accelerated step is adopted. Our experimental results show that the proposed method leads to excellent performances in terms of an objective evaluation and the subjective visual effect.

Structure Adaptive Total Variation Minimization-Based Image Decomposition

Structure-preserving image decomposition separates a given image into structure and texture by smoothing the image, simultaneously preserving or enhancing image edges. The well-studied problem of image decomposition is applied to various areas, such as image smoothing, detail enhancement, non-photorealistic rendering, image artistic rendering, and high-dynamic-range compression. In this paper, we propose a fast algorithm for structure-preserving image decomposition that adopts total variation (TV) minimization to the moving least squares (MLS) method with non-local weights, called structure adaptive TV (SATV) minimization. MLS with non-local weights provides high accuracy approximation that is robust to noise, and allows a fast convergence with TV regularization term. As a result, our proposed SATV preserves the dominant structure while flattening fine-scale details. The experimental results show that the SATV minimization algorithm provides faster and more robust image decomposition than the well-known previous approaches. We demonstrate the usefulness of our algorithm by presenting successful applications in image smoothing and detail enhancement.

Rolling bilateral filter-based text image deblurring

Although many competitive approaches have been developed for image deblurring, the priors which rely on natural image are less effective for text images which have special properties, lack of heavy-tailed gradient, and clean background. Considering the specific structure of text images, in this work, we present an effective yet simple deblurring method based on rolling bilateral filtering, an improved rolling guidance filtering specifically. The rolling bilateral filter can remove texture and preserve image structures. Thus, it is appropriate for processing the document image whose background regions are uniform. According to this property, we propose an efficient iterative algorithm to estimate the image and point spread function. Experimental results demonstrate that our method outperforms the traditional deblurring algorithms designed for typical natural images and text images.

Joint sparse reconstruction of multi-contrast MRI images with graph based redundant wavelet transform

BackgroundMulti-contrast images in magnetic resonance imaging (MRI) provide abundant contrast information reflecting the characteristics of the internal tissues of human bodies, and thus have been widely utilized in clinical diagnosis. However, long acquisition time limits the application of multi-contrast MRI. One efficient way to accelerate data acquisition is to under-sample the k-space data and then reconstruct images with sparsity constraint. However, images are compromised at high acceleration factor if images are reconstructed individually. We aim to improve the images with a jointly sparse reconstruction and Graph-based redundant wavelet transform (GBRWT).MethodsFirst, a sparsifying transform, GBRWT, is trained to reflect the similarity of tissue structures in multi-contrast images. Second, joint multi-contrast image reconstruction is formulated as a ℓ2, 1 norm optimization problem under GBRWT representations. Third, the optimization problem is numerically solved using a derived alternating direction method.ResultsExperimental results in synthetic and in vivo MRI data demonstrate that the proposed joint reconstruction method can achieve lower reconstruction errors and better preserve image structures than the compared joint reconstruction methods. Besides, the proposed method outperforms single image reconstruction with joint sparsity constraint of multi-contrast images.ConclusionsThe proposed method explores the joint sparsity of multi-contrast MRI images under graph-based redundant wavelet transform and realizes joint sparse reconstruction of multi-contrast images. Experiment demonstrate that the proposed method outperforms the compared joint reconstruction methods as well as individual reconstructions. With this high quality image reconstruction method, it is possible to achieve the high acceleration factors by exploring the complementary information provided by multi-contrast MRI.

Deep Semantic Structural Constraints for Zero-Shot Learning

Zero-shot learning aims to classify unseen image categories by learning a visual-semantic embedding space. In most cases, the traditional methods adopt a separated two-step pipeline that extracts image features are utilized to learn the embedding space. It leads to the lack of specific structural semantic information of image features for zero-shot learning task. In this paper, we propose an end-to-end trainable Deep Semantic Structural Constraints model to address this issue. The proposed model contains the Image Feature Structure constraint and the Semantic Embedding Structure constraint, which aim to learn structure-preserving image features and endue the learned embedding space with stronger generalization ability respectively. With the assistance of semantic structural information, the model gains more auxiliary clues for zero-shot learning. The state-of-the-art performance certifies the effectiveness of our proposed method.

Edge-Aware Spatial Denoising Filtering Based on a Psychological Model of Stimulus Similarity

Noise reduction is a fundamental operation in image quality enhancement. In recent years, a large body of techniques at the crossroads of statistics and functional analysis have been developed to minimize the blurring artifact introduced in the denoising process. Recent studies focus on edge-aware filters due to their tendency to preserve image structures. In this paper, we adopt a psychological model of similarity based on Shepard’s generalization law and introduce a new signal-dependent window selection technique. Such a focus is warranted, because blurring is essentially a cognitive act related to the human perception of physical stimuli (pixels). The proposed windowing technique can be used to implement a wide range of edge-aware spatial denoising filters, thereby transforming them into nonlocal filters. We employ simulations using both synthetic and real image samples to evaluate the performance of the proposed method by quantifying the enhancement in the signal strength, noise suppression, and structural preservation measured in terms of the peak signal-to-noise ratio (PSNR), mean square error (MSE), and structural similarity (SSIM) index, respectively. In our experiments, we observe that incorporating the proposed windowing technique in the design of mean, median, and nonlocal means filters substantially reduces the MSE while simultaneously increasing the PSNR and the SSIM.

Structure-Preserving Image Super-Resolution via Contextualized Multitask Learning

Single image super resolution (SR), which refers to reconstruct a higher-resolution (HR) image from the observed low-resolution (LR) image, has received substantial attention due to its tremendous application potentials. Despite the breakthroughs of recently proposed SR methods using convolutional neural networks (CNNs), their generated results usually lack of preserving structural (high-frequency) details. In this paper, regarding global boundary context and residual context as complimentary information for enhancing structural details in image restoration, we develop a contextualized multi-task learning framework to address the SR problem. Specifically, our method first extracts convolutional features from the input LR image and applies one deconvolutional module to interpolate the LR feature maps in a content-adaptive way. Then, the resulting feature maps are fed into two branched sub-networks. During the neural network training, one sub-network outputs salient image boundaries and the HR image, and the other sub-network outputs the local residual map, i.e., the residual difference between the generated HR image and ground-truth image. On several standard benchmarks (i.e., Set5, Set14 and BSD200), our extensive evaluations demonstrate the effectiveness of our SR method on achieving both higher restoration quality and computational efficiency compared with several state-of-the-art SR approaches. The source code and some SR results can be found at: http://hcp.sysu.edu.cn/structure-preserving-image-super-resolution/

Structure-preserving image completion with multi-level dynamic patches

In this paper, we present a novel structure-preserving image completion approach equipped with dynamic patches. We formulate the image completion problem into an energy minimization framework that accounts for coherence within the hole and global coherence simultaneously. The completion of the hole is achieved through iterative optimizations combined with a multi-scale solution. In order to avoid abnormal structure and disordered texture, we utilize a dynamic patch system to achieve efficient structure restoration. Our dynamic patch system functions in both horizontal and vertical directions of the image pyramid. In the horizontal direction, we conduct a parallel search for multi-size patches in each pyramid level and design a competitive mechanism to select the most suitable patch. In the vertical direction, we use large patches in higher pyramid level to maximize the structure restoration and use small patches in lower pyramid level to reduce computational workload. We test our approach on massive images with complex structure and texture. The results are visually pleasing and preserve nice structure. Apart from effective structure preservation, our approach outperforms previous state-of-the-art methods in time consumption.

Deep Convolutional Neural Network for Inverse Problems in Imaging.

In this paper, we propose a novel deep convolutional neural network (CNN)-based algorithm for solving ill-posed inverse problems. Regularized iterative algorithms have emerged as the standard approach to ill-posed inverse problems in the past few decades. These methods produce excellent results, but can be challenging to deploy in practice due to factors including the high computational cost of the forward and adjoint operators and the difficulty of hyperparameter selection. The starting point of this paper is the observation that unrolled iterative methods have the form of a CNN (filtering followed by pointwise nonlinearity) when the normal operator (H*H, where H* is the adjoint of the forward imaging operator, H) of the forward model is a convolution. Based on this observation, we propose using direct inversion followed by a CNN to solve normal-convolutional inverse problems. The direct inversion encapsulates the physical model of the system, but leads to artifacts when the problem is ill posed; the CNN combines multiresolution decomposition and residual learning in order to learn to remove these artifacts while preserving image structure. We demonstrate the performance of the proposed network in sparse-view reconstruction (down to 50 views) on parallel beam X-ray computed tomography in synthetic phantoms as well as in real experimental sinograms. The proposed network outperforms total variation-regularized iterative reconstruction for the more realistic phantoms and requires less than a second to reconstruct a 512 × 512 image on the GPU.

Weighted aggregation and fuzzy-concept-guided signal resemblance and expansion for video format conversion

In this paper, we propose an efficient deinterlacing method for HDTV that preserves image structures, edges, and details. In the human visual system, the eyes are more sensitive to high-frequency information such as edge details than low-frequency information such as image background. Therefore, averaging low-pass filter results is not effective for image enhancement. The proposed method is a weighted filtering approach that generates a half-pixel 9-by-9 edge-based line average window. We also propose pixel-resemblance- and pixel-expansion-based fuzzy weights, which are assigned using a triangular membership function. Compared to conventional format conversion methods, the proposed method outperforms all benchmarks in terms of both objective and subjective qualities.

Striped-texture image segmentation with application to multimedia security

In today’s rapid growth of volume of multimedia data, security is important yet challenging problem in multimedia applications. Image, which covers the highest percentage of the multimedia data, it is very important for multimedia security. Image segmentation is utilized as a fundamental preprocessing of various multimedia applications such as surveillance for security by breaking a given image into multiple salient regions. In this paper, we present a new image segmentation approach based on frequency-domain filtering for images with stripe texture, and generalize it to lattice fence images. Our method significantly reduces the impact of stripes on segmentation performance. The approach proposed in this paper consists of three phases. Given the images, we weaken the effect of stripe texture by filtering in the frequency domain automatically. Then, structure-preserving image smoothing is employed to remove texture details and extract the main image structures. Last, we use an effective threshold method to produce segmentation results. Our method achieves very promising results for the test image dataset and could benefit a number of new multimedia applications such as public security.

Scale-aware Structure-Preserving Texture Filtering

This paper presents a novel method to enhance the performance of structure-preserving image and texture filtering. With conventional edge-aware filters, it is often challenging to handle images of ...