Nonlocal Regularization Term Research Articles

HDR video synthesis by a nonlocal regularization variational model

High dynamic range (HDR) video synthesis is a very challenging task. Consecutive frames are acquired with alternate expositions, generally two or three different exposure times. Classical methods aim at registering neighboring frames and fuse them using image HDR techniques. However, the registration often fails to obtain accurate results and the fusion produces ghosting artifacts. Deep learning techniques have recently appeared imitating the structure of existing classical methods. The neural network is intended to estimate the registration function and choose the fusion weights. In this paper, we propose a new method for HDR video synthesis using a variational model. The proposed model uses a nonlocal regularization term to combine pixel information from neighboring frames. The obtained results are competitive with state-of-the-art. Moreover, the proposed method gives a more reliable and understandable solution than deep-learning based ones.

Inverse Halftoning Based on Weighted Nuclear Norm Minimization

The inverse halftoning technology refers to restoring a continuous-toned image from a halftoned image with only bi-level pixels. However, recovering the continuous images from their halftoned ones is normally ill-posed, which makes the inverse halftoning algorithm very challenging. In this paper, we propose an optimization model with two alternate projections (TAP) for image inverse halftoning under the weighted nuclear norm minimization (WNNM) framework. The main contributions are twofold. First, the WNNM nonlocal regularization term is established, which offers a powerful mechanism of nonlocal self-similarity to ensure a more reliable estimation. Second, the alternate minimization projections are formulated for solving the image inverse halftoning, which reconstructs the continuous-toned image without destroying the image details and structures. The experimental results showed that the proposed method outperformed the state of the arts in terms of both objective measurements and subjective visual performance. The codes and constructed models are available at: https://github.com/juneryoung2022/IH-WNNM.

Variational Fusion of Hyperspectral Data by Non-Local Filtering

The fusion of multisensor data has attracted a lot of attention in computer vision, particularly among the remote sensing community. Hyperspectral image fusion consists in merging the spectral information of a hyperspectral image with the geometry of a multispectral one in order to infer an image with high spatial and spectral resolutions. In this paper, we propose a variational fusion model with a nonlocal regularization term that encodes patch-based filtering conditioned to the geometry of the multispectral data. We further incorporate a radiometric constraint that injects the high frequencies of the scene into the fused product with a band per band modulation according to the energy levels of the multispectral and hyperspectral images. The proposed approach proved robust to noise and aliasing. The experimental results demonstrate the performance of our method with respect to the state-of-the-art techniques on data acquired by commercial hyperspectral cameras and Earth observation satellites.

Brain tissue segmentation via non-local fuzzy c-means clustering combined with Markov random field.

The segmentation and extraction of brain tissue in magnetic resonance imaging (MRI) is a meaningful task because it provides a diagnosis and treatment basis for observing brain tissue development, delineating lesions, and planning surgery. However, MRI images are often damaged by factors such as noise, low contrast and intensity brightness, which seriously affect the accuracy of segmentation. A non-local fuzzy c-means clustering framework incorporating the Markov random field for brain tissue segmentation is proposed in this paper. Firstly, according to the statistical characteristics that MRF can effectively describe the local spatial correlation of an image, a new distance metric with neighborhood constraints is constructed by combining probabilistic statistical information. Secondly, a non-local regularization term is integrated into the objective function to utilize the global structure feature of the image, so that both the local and global information of the image can be taken into account. In addition, a linear model of inhomogeneous intensity is also built to estimate the bias field in brain MRI, which has achieved the goal of overcoming the intensity inhomogeneity. The proposed model fully considers the randomness and fuzziness in the image segmentation problem, and obtains the prior knowledge of the image reasonably, which reduces the influence of low contrast in the MRI images. Then the experimental results demonstrate that the proposed method can eliminate the noise and intensity inhomogeneity of the MRI image and effectively improve the image segmentation accuracy.

Coupled damage variable based on fracture locus: Modelling and calibration

A continuum ductile damage and failure model coupled with metal plasticity is presented, with the focus on capturing different failure mechanisms and prediction of the strain localisation. The onset of diffuse necking and local thinning is observed in the early stage of experiment and it is caused by the very low strain hardening capability of the material under study. Uniaxial tensile tests have been performed on notched and shear samples, specifically designed to cover different stress states by varying both the notch radius and orientation with respect to the sample axis. The experiments have revealed strain localisation caused by material softening. The latter is modelled by a coupled damage accumulation rule whose magnitude is dictated by the actual stress state and strain to fracture. Since the numerical model relies on the fracture strain determined by digital image correlation (DIC), which is dependent on the length scale imposed by the resolution of the DIC grid, an experimentally resolved length scale parameter l is introduced by nonlocal regularization of the material flow, based on large-deformation gradient theory. Apart from its common role of ensuring mesh independency, the length scale parameter obtains a new additional role: it makes the fracture locus calibration procedure robust under changes induced by different spatial averaging of the experimental fracture strain. The measured width of the strain localisation equals to the regularization length scale parameter l used in finite element (FE) simulations, thereby ensuring tight correspondence of physical reality and numerical model on the basis of a measurable quantity. As a result, the nonlocal regularization term prevents the strain in the numerical model to localise to a higher extent than experimentally observed. Eventually, the fracture locus has been constructed as a function of stress triaxiality ratio and Lode angle parameter. The FE-modelling methodology is calibrated by force-displacement curves and surface strain profiles prior to failure, obtained by DIC for a subset of sample geometries. The predictive capability of the proposed methodology is demonstrated by computing and comparing FE results to another experimentally characterized sample, not used in calibration procedure, i.e. dog-bone specimen. It undergoes a complicated loading path different from the paths used in the calibration procedure. It has been shown that regularization of strains and metal plasticity model supplemented by a damage variable are indispensable for an overall agreement of experimental and numerical results. The unique numerical model formulation has proven capable of capturing strongly different ductile failure modes, which are experimentally observed and further discussed.

Self-Similarity Sparse Representation for Image Restoration

Image restoration aims to restore an image from a degraded image. The degradation may occur during image acquisition or image transmission. Image degradation lowers the quality of the image. In this paper additive Gaussian noise is considered for degrading the original image. For restoring the image from degraded image the proposed method used both local and non-local similarity patterns. The restoration problem is modeled with regression model. Two regularization terms are considered for representing prior image information. One regularization term is for local patterns and other is for non-local similarity patterns. The additive local regularization term is used to restore the edges. The non-local regularization term works best for local smoothness and edge information will be lost. The proposed algorithm took a clean image of size 256x256 and added with Gaussian noise with different levels of noise levels. A self-adaptive dictionary is trained for a particular window of image with local and non-local patterns and stacked to three dimensional matrix. The patch size considered for training the dictionary is 10x10. For restoring each patch it searches best atoms form the trained dictionary. The efficiency of the algorithm is estimated by parameters mean square error, root mean square error, PSNR and FSIM. The algorithm is also tested for different images like cameraman, house, Barbara, Lena and parrot. The proposed algorithm is tested with conventional algorithms. .

A multiframe super-resolution technique based on a nonlocal Bregman distance of bilateral total variation term

In this paper, we propose a variational multiframe super-resolution (SR) model based on a nonlocal regularization term using Bregman distances. Since the SR algorithms always skip the complex spatial interactions within images, we introduce a nonlocal form of the bilateral total variation (BTV) regularization term which can take into consideration these interactions and also efficiently preserve strong edges and contours of the reconstructed high resolution (HR) image for our model. In addition, to avoid contrast loss and smoothing gray values in the SR process, we introduce Bregman distances which produce a more consistent model. Moreover, to resolve the obtained SR algorithm, we use a first-order primal dual algorithm to ensure that the convergence to the desired HR image can be achieved in a fast way. As a result, the proposed algorithm shows improved performance compared to other SR methods.

A survey of pansharpening methods with a new band-decoupled variational model

Most satellites decouple the acquisition of a panchromatic image at high spatial resolution from the acquisition of a multispectral image at lower spatial resolution. Pansharpening is a fusion technique used to increase the spatial resolution of the multispectral data while simultaneously preserving its spectral information. In this paper, we consider pansharpening as an optimization problem minimizing a cost function with a nonlocal regularization term. The energy functional which is to be minimized decouples for each band, thus permitting the application to misregistered spectral components. This requirement is achieved by dropping the, commonly used, assumption that relates the spectral and panchromatic modalities by a linear transformation. Instead, a new constraint that preserves the radiometric ratio between the panchromatic and each spectral component is introduced. An exhaustive performance comparison of the proposed fusion method with several classical and state-of-the-art pansharpening techniques illustrates its superiority in preserving spatial details, reducing color distortions, and avoiding the creation of aliasing artifacts.

Video restoration based on a novel second order nonlocal total variation model

Total variation and its variants have been widely used in the video/image restoration area in the past decades. Among them, the nonlocal total variation model introduces penalization on nonlocal gradients and demonstrates remarkable performance gain in many applications. However, this approach tends to suppress intensity-changes of visual contents, and hence cannot restore complicated visual contents well enough. To address this issue, this paper proposes a novel second order nonlocal total variation model for video restoration problems. Firstly, the directed space-time nonlocal gradients are defined in our model to formulate the spatio-temporal intensity-changes of video contents. Secondly, mean-value approximations of these nonlocal gradients are introduced into the model. Based on them, we build up a new model that consists of both first order and second order nonlocal regularization terms. Furthermore, for the purpose of adapting to specific visual contents, it is also augmented with a content-adaptive modification. These features make the proposed second order nonlocal total variation model a high order generalization of the original one. Experimental results on various video restoration problems show that the proposed model significantly improves the restoration qualities, compared with other state-of-the-art approaches.

Image restoration approach using a joint sparse representation in 3D-transform domain

Image restoration is a crucial problem in image processing and a necessary step before the image segmentation and recognition. A new framework for image restoration in 3D transform domain terms as joint sparse representation (JSR) is proposed in this work. The proposed JSR is able to represent image more sparsely and more precisely in the transform domain by performing 3D transform on each set of similar blocks. In addition to that, in order to overcome the issues of defective block matching and spurious artifact in the 3D sparse representation, JSR introduces a new nonlocal regularization term which characterizes the statistics of the nonlocal image to improve the accuracy of the estimated coefficients. The parameters of regularization terms are calculated based on Bayesian philosophy, and a split Bregman-based technique is developed to obtain the solution in a tractable and robust manner. Extensive experiments on image denoising, image inpainting and image deblurring demonstrate that the proposed JSR algorithm outperforms current state-of-the-art approaches in terms of peak signal-to-noise ratio and visual quality.

Poissonian Hyperspectral Image Superresolution Using Alternating Direction Optimization

The reconstruction of Poissonian image is an active research area in recent years. This paper proposes a novel method for Poissonian hyperspectral image superresolution by fusing a low-spatial-resolution hyperspectral image and a low-spectral-resolution multispectral image. The fusion scheme is designed as an optimization problem, whose cost function consists of the two data-fidelity terms about Poisson distribution, the sparse representation term, and the nonlocal regularization term. The two data-fidelity terms can capture statistical information of Poisson noise. The sparse representation term is used for enhancing the quality of sparsity-based signal reconstruction, and the nonlocal regularization term exploits the spatial similarity of hyperspectral image. As a result, the hyperspectral image and multispectral image are well fused. Finally, the designed optimization problem is effectively solved by an alternating direction optimization algorithm. Simulation results illustrate that the proposed method has a better performance than several well-known methods both in terms of quality indexes and reconstruction visual effect.

Temporal sparsity exploiting nonlocal regularization for 4D computed tomography reconstruction.

X-ray imaging applications in medical and material sciences are frequently limited by the number of tomographic projections collected. The inversion of the limited projection data is an ill-posed problem and needs regularization. Traditional spatial regularization is not well adapted to the dynamic nature of time-lapse tomography since it discards the redundancy of the temporal information. In this paper, we propose a novel iterative reconstruction algorithm with a nonlocal regularization term to account for time-evolving datasets. The aim of the proposed nonlocal penalty is to collect the maximum relevant information in the spatial and temporal domains. With the proposed sparsity seeking approach in the temporal space, the computational complexity of the classical nonlocal regularizer is substantially reduced (at least by one order of magnitude). The presented reconstruction method can be directly applied to various big data 4D (x, y, z+time) tomographic experiments in many fields. We apply the proposed technique to modelled data and to real dynamic X-ray microtomography (XMT) data of high resolution. Compared to the classical spatio-temporal nonlocal regularization approach, the proposed method delivers reconstructed images of improved resolution and higher contrast while remaining significantly less computationally demanding.

Improved image reconstruction of low-resolution multichannel phase contrast angiography.

In low-resolution phase contrast magnetic resonance angiography, the maximum intensity projected channel images will be blurred with consequent loss of vascular details. The channel images are enhanced using a stabilized deblurring filter, applied to each channel prior to combining the individual channel images. The stabilized deblurring is obtained by the addition of a nonlocal regularization term to the reverse heat equation, referred to as nonlocally stabilized reverse diffusion filter. Unlike reverse diffusion filter, which is highly unstable and blows up noise, nonlocal stabilization enhances intensity projected parallel images uniformly. Application to multichannel vessel enhancement is illustrated using both volunteer data and simulated multichannel angiograms. Robustness of the filter applied to volunteer datasets is shown using statistically validated improvement in flow quantification. Improved performance in terms of preserving vascular structures and phased array reconstruction in both simulated and real data is demonstrated using structureness measure and contrast ratio.

Graph-based Sparse Representation for Image Denoising

Sparse representation has shown the effectiveness in solving image restoration and classification problems. To improve the performance of sparse representation, the patch-based and graph-based regularization term with respect to the sparse coding are proposed to solve image restoration and classification problems, respectively. In this paper, the local manifold structure of intensity on patches is exploited by a graph Laplacian operator for performing more precise estimation of sparse coding. Additionally, an improved nonlocal regularization term with the local manifold structure information is proposed to preserve the texture more effectively compared with the traditional nonlocal regularization term. The experimental results on image denoising show the promising performance in terms of both peak signal noise ratio and visual perception.

Depth map super-resolution via local and nonlocal priors

We propose a method for solving the depth map super-resolution problem. Given a low-resolution depth map as input, we recover a high-resolution depth map using a registered and potentially high-resolution color image. We formulate the super-resolution process as a regularization and optimization framework. Specifically, taking into account that discontinuities in depth and color tend to coalign, we obtain the local and nonlocal regularization terms based on the raw depth map and also the features of high-resolution color images. Different from conventional methods, we model the relationship with two constraint terms of local and nonlocal priors to sufficiently explore the complementary nature. Experimental results demonstrate the effectiveness of our approach, which produces sharper edges and more faithful details compared with other state-of-the-art strategies.

Implementation of Nonlocal Pansharpening Image Fusion

This paper focuses on the implementation of the pansharpened image fusion technique proposed in the companion paper [A Nonlocal Variational Model for Pansharpening Image Fusion, SIAM Journal on Imaging Sciences, 2014, to appear]. Pansharpening refers to the process of inferring a high resolution multispectral image from a high resolution panchromatic image and low resolution multispectral one. Although quite successful in terms of relative error, state-of-the-art pansharpening methods still introduce relevant color artifacts. The variational pansharpening model proposed by Buades et al. incorporates a nonlocal regularization term that takes advantage of image self-similarity, leading to signicant reduction of the above-mentioned color artifacts.

A Nonlocal Variational Model for Pansharpening Image Fusion

Pansharpening refers to the fusion process of inferring a high-resolution multispectral image from a high-resolution panchromatic image and a low-resolution multispectral one. In this paper we propose a new variational method for pansharpening which incorporates a nonlocal regularization term and two fidelity terms, one describing the relation between the panchromatic image and the high-resolution spectral channels and the other one preserving the colors from the low-resolution modality. The nonlocal term is based on the image self-similarity principle applied to the panchromatic image. The existence and uniqueness of minimizer for the described functional is proved in a suitable space of weighted integrable functions. Although quite successful in terms of relative error, state-of-the-art pansharpening methods introduce relevant color artifacts. These spectral distortions can be significantly reduced by involving the image self-similarity. Extensive comparisons with state-of-the-art algorithms are performed.

A nonlocal energy minimization approach to brain image segmentation with simultaneous bias field estimation and denoising

Image segmentation plays an important role in medical image analysis. The most widely used image segmentation algorithms, region-based methods that typically rely on the homogeneity of image intensities in the regions of interest, often fail to provide accurate segmentation results due to the existence of bias field, heavy noise and rich structures. In this paper, we incorporate nonlocal regularization mechanism in the coherent local intensity clustering formulation for brain image segmentation with simultaneously estimating bias field and denoising, specially preserving good structures. We define an energy functional with a local data fitting term, two nonlocal regularization terms for both image and membership functions, and a $$L_2$$ L 2 image fidelity term. By minimizing the energy, we get good segmentation results with well preserved structures. Meanwhile, the bias estimation and noise reduction can also be achieved. Experiments performed on synthetic and clinical brain magnetic resonance imaging data and comparisons with other methods are given to demonstrate that by introducing the nonlocal regularization mechanism, we can get more regularized segmentation results.

On level set regularization for highly ill-posed distributed parameter estimation problems

The recovery of a distributed parameter function with discontinuities from inverse problems with elliptic forward PDEs is fraught with theoretical and practical difficulties. Better results are obtained for problems where the solution may take on at each point only one of two values, thus yielding a shape recovery problem. This article considers level set regularization for such problems. However, rather than explicitly integrating a time embedded PDE to steady state, which typically requires thousands of iterations, methods based on Gauss–Newton are applied more directly. One of these can be viewed as damped Gauss–Newton utilized to approximate the steady state equations which in turn are viewed as the necessary conditions of a Tikhonov-type regularization with a sharpening sub-step at each iteration. In practice this method is eclipsed, however, by a special “finite time” or Levenberg–Marquardt-type method which we call dynamic regularization applied to the output least squares formulation. Our stopping criterion for the iteration does not involve knowledge of the true solution. The regularization functional is applied to the (smooth) level set function rather than to the discontinuous function to be recovered, and the second focus of this article is on selecting this functional. Typical choices may lead to flat level sets that in turn cause ill-conditioning. We propose a new, quartic, non-local regularization term that penalizes flatness and produces a smooth level set evolution, and compare its performance to more usual choices. Two numerical test cases are considered: a potential problem and the classical EIT/DC resistivity problem.