Image Nonlocal Self-similarity Research Articles

Joint group dictionary-based structural sparse representation for image restoration

With the aid of image nonlocal self-similarity (NSS), recent studies have indicated that structural sparse representation (SSR) models with patch groups as processing units have a great potential in various image inverse problems. Most of existing works, however, only use the dictionary learned from internal or external patch groups for sparse representation. Consequently, such methods generally overfit degradation or do not adapt to the given image. In this paper, we propose a novel sparse representation model, termed joint group dictionary-based SSR (JGD-SSR) model, to alleviate the above problem. Compared with traditional SSR models, the proposed JGD-SSR provides an effective mechanism to exploit internal and external dictionaries jointly. Moreover, to make the proposed model more robust, the connection between JGD-SSR and maximum a posteriori (MAP) estimation is revealed to adaptively estimate regularization parameters. When we apply the proposed JGD-SSR to image restoration tasks, an alternating minimization scheme is developed to solve the resulting optimization problems. The experimental results, including image denoising, deblocking, and deblurring, demonstrate that the proposed JGD-SSR-based restoration algorithm outperforms many state-of-the-art competing methods in terms of both objective and visual qualities.

A Hybrid Structural Sparsification Error Model for Image Restoration.

Recent works on structural sparse representation (SSR), which exploit image nonlocal self-similarity (NSS) prior by grouping similar patches for processing, have demonstrated promising performance in various image restoration applications. However, conventional SSR-based image restoration methods directly fit the dictionaries or transforms to the internal (corrupted) image data. The trained internal models inevitably suffer from overfitting to data corruption, thus generating the degraded restoration results. In this article, we propose a novel hybrid structural sparsification error (HSSE) model for image restoration, which jointly exploits image NSS prior using both the internal and external image data that provide complementary information. Furthermore, we propose a general image restoration scheme based on the HSSE model, and an alternating minimization algorithm for a range of image restoration applications, including image inpainting, image compressive sensing and image deblocking. Extensive experiments are conducted to demonstrate that the proposed HSSE-based scheme outperforms many popular or state-of-the-art image restoration methods in terms of both objective metrics and visual perception.

A Hybrid Sparse Representation Model for Image Restoration.

Group-based sparse representation (GSR) uses image nonlocal self-similarity (NSS) prior to grouping similar image patches, and then performs sparse representation. However, the traditional GSR model restores the image by training degraded images, which leads to the inevitable over-fitting of the data in the training model, resulting in poor image restoration results. In this paper, we propose a new hybrid sparse representation model (HSR) for image restoration. The proposed HSR model is improved in two aspects. On the one hand, the proposed HSR model exploits the NSS priors of both degraded images and external image datasets, making the model complementary in feature space and the plane. On the other hand, we introduce a joint sparse representation model to make better use of local sparsity and NSS characteristics of the images. This joint model integrates the patch-based sparse representation (PSR) model and GSR model, while retaining the advantages of the GSR model and the PSR model, so that the sparse representation model is unified. Extensive experimental results show that the proposed hybrid model outperforms several existing image recovery algorithms in both objective and subjective evaluations.

Non-Local Robust Quaternion Matrix Completion for Large-Scale Color Image and Video Inpainting.

The image nonlocal self-similarity (NSS) prior refers to the fact that a local patch often has many nonlocal similar patches to it across the image and has been widely applied in many recently proposed machining learning algorithms for image processing. However, there is no theoretical analysis on its working principle in the literature. In this paper, we discover a potential causality between NSS and low-rank property of color images, which is also available to grey images. A new patch group based NSS prior scheme is proposed to learn explicit NSS models of natural color images. The numerical low-rank property of patched matrices is also rigorously proved. The NSS-based QMC algorithm computes an optimal low-rank approximation to the high-rank color image, resulting in high PSNR and SSIM measures and particularly the better visual quality. A new tensor NSS-based QMC method is also presented to solve the color video inpainting problem based on quaternion tensor representation. The numerical experiments on color images and videos indicate the advantages of NSS-based QMC over the state-of-the-art methods.

Image Restoration via Reconciliation of Group Sparsity and Low-Rank Models.

Image nonlocal self-similarity (NSS) property has been widely exploited via various sparsity models such as joint sparsity (JS) and group sparse coding (GSC). However, the existing NSS-based sparsity models are either too restrictive, e.g., JS enforces the sparse codes to share the same support, or too general, e.g., GSC imposes only plain sparsity on the group coefficients, which limit their effectiveness for modeling real images. In this paper, we propose a novel NSS-based sparsity model, namely, low-rank regularized group sparse coding (LR-GSC), to bridge the gap between the popular GSC and JS. The proposed LR-GSC model simultaneously exploits the sparsity and low-rankness of the dictionary-domain coefficients for each group of similar patches. An alternating minimization with an adaptive adjusted parameter strategy is developed to solve the proposed optimization problem for different image restoration tasks, including image denoising, image deblocking, image inpainting, and image compressive sensing. Extensive experimental results demonstrate that the proposed LR-GSC algorithm outperforms many popular or state-of-the-art methods in terms of objective and perceptual metrics.

Fast Non-Local Adaptive In-Loop Filter Optimization on GPU

The non-local adaptive in-loop filter (NALF) for video coding has achieved significant coding gain by exploiting image non-local self-similarity (NSS) to efficiently reduce the compression artifacts. However, the intensive computation of NALF hinders its practical deployment in video standardizations. In this paper, we propose a fast NALF optimization algorithm in parallel-computing framework by leveraging the massive parallel execution resources of GPU. First, the computational complexity of original NALF is analyzed in depth, then the pipelines of computational-intensive modules are re-designed to adapt to the general-purpose GPU with more parallel-friendly consideration. Specifically, we speed up the NALF by optimizing thread allocation to maximize the parallelism degree and elaborately designing the GPU block dimension to avoid access conflict. The group-level and pixel-level parallelization for collaboratively filtering and patch matching modules are designed respectively. To reduce the cost in data transmission, the whole filtering process is implemented on GPU by taking the advantage of low data dependency in NALF. Extensive experimental results show that the proposed fast NALF optimization using GPU architecture achieves high-speeed processing while maintaining the significant coding performance of original NALF, which shows the potential of NALF in the future video coding standard.

Group Sparsity Residual Constraint with Non-Local Priors for Image Restoration.

Group sparse representation (GSR) has made great strides in image restoration producing superior performance, realized through employing a powerful mechanism to integrate the local sparsity and nonlocal self-similarity of images. However, due to some form of degradation (e.g., noise, down-sampling or pixels missing), traditional GSR models may fail to faithfully estimate sparsity of each group in an image, thus resulting in a distorted reconstruction of the original image. This motivates us to design a simple yet effective model that aims to address the above mentioned problem. Specifically, we propose group sparsity residual constraint with nonlocal priors (GSRC-NLP) for image restoration. Through introducing the group sparsity residual constraint, the problem of image restoration is further defined and simplified through attempts at reducing the group sparsity residual. Towards this end, we first obtain a good estimation of the group sparse coefficient of each original image group by exploiting the image nonlocal self-similarity (NSS) prior along with self-supervised learning scheme, and then the group sparse coefficient of the corresponding degraded image group is enforced to approximate the estimation. To make the proposed scheme tractable and robust, two algorithms, i.e., iterative shrinkage/thresholding (IST) and alternating direction method of multipliers (ADMM), are employed to solve the proposed optimization problems for different image restoration tasks. Experimental results on image denoising, image inpainting and image compressive sensing (CS) recovery, demonstrate that the proposed GSRC-NLP based image restoration algorithm is comparable to state-of-the-art denoising methods and outperforms several state-of-the-art image inpainting and image CS recovery methods in terms of both objective and perceptual quality metrics.

Image Restoration via Simultaneous Nonlocal Self-Similarity Priors.

Through exploiting the image nonlocal self-similarity (NSS) prior by clustering similar patches to construct patch groups, recent studies have revealed that structural sparse representation (SSR) models can achieve promising performance in various image restoration tasks. However, most existing SSR methods only exploit the NSS prior from the input degraded (internal) image, and few methods utilize the NSS prior from external clean image corpus; how to jointly exploit the NSS priors of internal image and external clean image corpus is still an open problem. In this paper, we propose a novel approach for image restoration by simultaneously considering internal and external nonlocal self-similarity (SNSS) priors that offer mutually complementary information. Specifically, we first group nonlocal similar patches from images of a training corpus. Then a group-based Gaussian mixture model (GMM) learning algorithm is applied to learn an external NSS prior. We exploit the SSR model by integrating the NSS priors of both internal and external image data. An alternating minimization with an adaptive parameter adjusting strategy is developed to solve the proposed SNSS-based image restoration problems, which makes the entire algorithm more stable and practical. Experimental results on three image restoration applications, namely image denoising, deblocking and deblurring, demonstrate that the proposed SNSS produces superior results compared to many popular or state-of-the-art methods in both objective and perceptual quality measurements.

From Rank Estimation to Rank Approximation: Rank Residual Constraint for Image Restoration.

In this paper, we propose a novel approach for the rank minimization problem, termed rank residual constraint (RRC). Different from existing low-rank based approaches, such as the well-known nuclear norm minimization (NNM) and the weighted nuclear norm minimization (WNNM), which estimate the underlying low-rank matrix directly from the corrupted observation, we progressively approximate (approach) the underlying low-rank matrix via minimizing the rank residual. Through integrating the image nonlocal self-similarity (NSS) prior with the proposed RRC model, we apply it to image restoration tasks, including image denoising and image compression artifacts reduction. Toward this end, we first obtain a good reference of the original image groups by using the image NSS prior, and then the rank residual of the image groups between this reference and the degraded image is minimized to achieve a better estimate to the desired image. In this manner, both the reference and the estimated image in each iteration are improved gradually and jointly. Based on the group-based sparse representation model, we further provide a theoretical analysis on the feasibility of the proposed RRC model. Experimental results demonstrate that the proposed RRC model outperforms many state-of-the-art schemes in both the objective and perceptual qualities.

Improved Image Denoising Algorithm Based on Superpixel Clustering and Sparse Representation

Good learning image priors from the noise-corrupted images or clean natural images are very important in preserving the local edge and texture regions while denoising images. This paper presents a novel image denoising algorithm based on superpixel clustering and sparse representation, named as the superpixel clustering and sparse representation (SC-SR) algorithm. In contrast to most existing methods, the proposed algorithm further learns image nonlocal self-similarity (NSS) prior with mid-level visual cues via superpixel clustering by the sparse subspace clustering method. As the superpixel edges adhered to the image edges and reflected the image structural features, structural and edge priors were considered for a better exploration of the NSS prior. Next, each similar superpixel region was regarded as a searching window to seek the first L most similar patches to each local patch within it. For each similar superpixel region, a specific dictionary was learned to obtain the initial sparse coefficient of each patch. Moreover, to promote the effectiveness of the sparse coefficient for each patch, a weighted sparse coding model was constructed under a constraint of weighted average sparse coefficient of the first L most similar patches. Experimental results demonstrated that the proposed algorithm achieved very competitive denoising performance, especially in image edges and fine structure preservation in comparison with state-of-the-art denoising algorithms.

Consistent Coding Scheme for Single-Image Super-Resolution Via Independent Dictionaries

In this paper, we present a unified frame based on collaborative representation (CR) for single-image super-resolution (SR), which learns low-resolution (LR) and high-resolution (HR) dictionaries independently in the training stage and adopts a consistent coding scheme (CCS) to guarantee the prediction accuracy of HR coding coefficients during SR reconstruction. The independent LR and HR dictionaries are learned based on CR with $l_{2}$ -norm regularization, which can well describe the corresponding LR and HR patch space, respectively. Furthermore, a mapping function is learned to map LR coding coefficients onto the corresponding HR coding coefficients. Propagation filtering can achieve smoothing over an image while preserving image context like edges or textural regions. Moreover, to preserve the edge structures of a super-resolved image and suppress artifacts, a propagation filtering-based constraint and image nonlocal self-similarity regularization are introduced into the SR reconstruction framework. Experimental comparison with state-of-the-art single image SR algorithms validates the effectiveness of proposed approach.

Wavelet-NARM Based Sparse Representation for Bio Medical Images

Sparse representation based super resolution deals with the problem of reconstructing a high resolution image from one or several of its low resolution counterparts. In this case the low resolution image is modelled as the down-sampled version of its high resolution counterpart after blurring. When the blurring kernel is the Dirac delta function, i.e. the low resolution image is directly down sampled from its high resolution counterpart without blurring and the super-resolution problem becomes an image interpolation problem. In such cases, the conventional sparse representation models become less effective, because the data fidelity term fails to constrain the image local structures. In natural images, the given image patch can be modelled as the linear combination of nonlocal similar neighbours. In this paper image nonlocal self-similarity for image interpolation is introduced. More specifically, wavelet based a nonlocal autoregressive model (NARM) is proposed and taken as the data fidelity term in sparse representation model. Our experimental results on benchmark test images clearly demonstrate that the proposed wavelet-NARM based image interpolation method outperforms the reconstruction of edge structures and suppression of jaggy/ringing artefacts, achieving the best image interpolation results so far in terms of PSNR as well as perceptual quality metrics such as structural similarity index and structural content. The proposed method is applied on bio medical images to emphasis on diagnostic information.

Hyperspectral Image Classification Based on Nonlocal Means With a Novel Class-Relativity Measurement

Nonlocal means (NLM) algorithm has been proven to be an effective context-sensitive denoising approach, where many similar patches spatially far from a given patch could provide nonlocal constraint to the local structure. For hyperspectral image, however, the conventional NLM algorithm becomes inapplicable for the high number of spectral bands. In this letter, we incorporate the image nonlocal self-similarity into the maximum a posteriori estimation for hyperspectral classification. The main novelty lies in the following two aspects: The NLM algorithm is exploited to combine similar local structures and nonlocal averaging; a new class-relativity measurement is proposed to describe the self-similarity in the context of the hyperspectral classification. Several experiments on simulated and real hyperspectral data sets are provided to demonstrate the effectiveness of the proposed algorithm.

Sparse Representation Based Image Interpolation With Nonlocal Autoregressive Modeling

Sparse representation is proven to be a promising approach to image super-resolution, where the low-resolution (LR) image is usually modeled as the down-sampled version of its high-resolution (HR) counterpart after blurring. When the blurring kernel is the Dirac delta function, i.e., the LR image is directly down-sampled from its HR counterpart without blurring, the super-resolution problem becomes an image interpolation problem. In such cases, however, the conventional sparse representation models (SRM) become less effective, because the data fidelity term fails to constrain the image local structures. In natural images, fortunately, many nonlocal similar patches to a given patch could provide nonlocal constraint to the local structure. In this paper, we incorporate the image nonlocal self-similarity into SRM for image interpolation. More specifically, a nonlocal autoregressive model (NARM) is proposed and taken as the data fidelity term in SRM. We show that the NARM-induced sampling matrix is less coherent with the representation dictionary, and consequently makes SRM more effective for image interpolation. Our extensive experimental results demonstrate that the proposed NARM-based image interpolation method can effectively reconstruct the edge structures and suppress the jaggy/ringing artifacts, achieving the best image interpolation results so far in terms of PSNR as well as perceptual quality metrics such as SSIM and FSIM.

Nonlocally Centralized Sparse Representation for Image Restoration

Sparse representation models code an image patch as a linear combination of a few atoms chosen out from an over-complete dictionary, and they have shown promising results in various image restoration applications. However, due to the degradation of the observed image (e.g., noisy, blurred, and/or down-sampled), the sparse representations by conventional models may not be accurate enough for a faithful reconstruction of the original image. To improve the performance of sparse representation-based image restoration, in this paper the concept of sparse coding noise is introduced, and the goal of image restoration turns to how to suppress the sparse coding noise. To this end, we exploit the image nonlocal self-similarity to obtain good estimates of the sparse coding coefficients of the original image, and then centralize the sparse coding coefficients of the observed image to those estimates. The so-called nonlocally centralized sparse representation (NCSR) model is as simple as the standard sparse representation model, while our extensive experiments on various types of image restoration problems, including denoising, deblurring and super-resolution, validate the generality and state-of-the-art performance of the proposed NCSR algorithm.

Image Deblurring and Super-Resolution by Adaptive Sparse Domain Selection and Adaptive Regularization

As a powerful statistical image modeling technique, sparse representation has been successfully used in various image restoration applications. The success of sparse representation owes to the development of the l(1)-norm optimization techniques and the fact that natural images are intrinsically sparse in some domains. The image restoration quality largely depends on whether the employed sparse domain can represent well the underlying image. Considering that the contents can vary significantly across different images or different patches in a single image, we propose to learn various sets of bases from a precollected dataset of example image patches, and then, for a given patch to be processed, one set of bases are adaptively selected to characterize the local sparse domain. We further introduce two adaptive regularization terms into the sparse representation framework. First, a set of autoregressive (AR) models are learned from the dataset of example image patches. The best fitted AR models to a given patch are adaptively selected to regularize the image local structures. Second, the image nonlocal self-similarity is introduced as another regularization term. In addition, the sparsity regularization parameter is adaptively estimated for better image restoration performance. Extensive experiments on image deblurring and super-resolution validate that by using adaptive sparse domain selection and adaptive regularization, the proposed method achieves much better results than many state-of-the-art algorithms in terms of both PSNR and visual perception.