Data Fidelity Term Research Articles

Deep Energy: Task Driven Training of Deep Neural Networks

The current gold standard in solving image processing and computer vision tasks is using supervised learning of deep neural networks (DNNs), requiring large-scale datasets of input-output pairs. In many scenarios in which the output is an image - e.g., medical image analysis, image denoising, deblurring, super-resolution, dehazing, segmentation and optical flow estimation - the collection of labelled image pairs for training is either time-consuming or limited to simple degradation models. Indeed, there is an increasing body of work targeted at weakly supervised training, accompanied with different unsupervised loss functions. This work dives into the regime of Deep-Energy, a task-driven training approach that substitutes the generic loss with minimization of energy functions using DNNs. Such energy functions, often formulated as a combination of a data-fidelity term along with an application-specific prior, are essentially unsupervised as they do not assume knowledge of the output image. As opposed to classic energy minimization, where computationally-intensive inference is performed for each new image, our network, once trained, can compute the output with a single forward-pass operation. By incorporating application-specific domain knowledge to the loss function, we are able to use real-world images, thus decreasing the dependency on pixel-wise labelled data or synthetic datasets. We demonstrate our approach on three different applications: seeded segmentation, image matting and single image dehazing, showing clear benefits of both speedup and improved accuracy versus the classical energy minimization approach, and competitive performance with respect to fully supervised alternatives.

Spatiotemporal road scene reconstruction using superpixel-based Markov random field

Scene reconstruction based on image rendering is an indispensable but challenging task in computer vision and intelligent transportation systems. We propose a framework for reconstructing road scenes as 3D corridor models and consisting of two stages: road detection and scene reconstruction. Road detection is realized by the novel superpixel-based Markov random field. The data fidelity term in the energy function is jointly computed according to superpixel features of color, texture and location. The smoothness term is established from the interaction of spatiotemporally adjacent superpixels. In the subsequent scene reconstruction, the foreground and background regions are modeled independently. Experiments on road detection demonstrate that our proposal outperforms state-of-the-art methods in both accuracy and computation speed. Scene reconstruction experiments further confirm that the scene models retrieved by the proposed method have higher correctness ratio, and can support several applications.

A parallel and automatically tuned algorithm for multispectral image deconvolution

ABSTRACT In the era of big data, radio astronomical image reconstruction algorithms are challenged to estimate clean images given limited computing resources and time. This article is driven by the need for large-scale image reconstruction for the future Square Kilometre Array (SKA), which will become in the next decades the largest low and intermediate frequency radio telescope in the world. This work proposes a scalable wide-band deconvolution algorithm called MUFFIN, which stands for ‘MUlti Frequency image reconstruction For radio INterferometry’. MUFFIN estimates the sky images in various frequency bands, given the corresponding dirty images and point spread functions. The reconstruction is achieved by minimizing a data fidelity term and joint spatial and spectral sparse analysis regularization terms. It is consequently non-parametric w.r.t. the spectral behaviour of radio sources. MUFFIN algorithm is endowed with a parallel implementation and an automatic tuning of the regularization parameters, making it scalable and well suited for big data applications such as SKA. Comparisons between MUFFIN and the state-of-the-art wide-band reconstruction algorithm are provided.

BM3D-based total variation algorithm for speckle removal with structure-preserving in OCT images.

In this paper, we propose a total variation based on block matching 3D (BM3D-TV method), which includes the total variation regular term, the data fidelity term, and the block matching term. In addition, we also propose a fast numerical algorithm based on the split Bregman iteration for the proposed method. By assigning suitable weights to the data fidelity term and block matching term, the image noise reduction and the image structural characteristics can be matched optimally. We test the proposed method on six human retinal and one mouse skin optical coherence tomography (OCT) images respectively, and also compare it with total variation (TV) and BM3D, which were proved to be effective in denoising. The performances of these methods are quantitatively evaluated in terms of the signal-to-noise ratio, the contrast-to-noise ratio, and the averaged equivalent number of homogeneous areas at the aspects of speckle reduction and structure protection. Vast experiments show that the BM3D-TV method can effectively reduce speckle noise in OCT images, protect important structural information and improve image quality, compared with the BM3D and TV methods.

A SAR Image Despeckling Method Using Multi-Scale Nonlocal Low-Rank Model

Speckle noise is an inherent nature of synthetic aperture radar (SAR) images, which degrades the quality of the images and makes the interpretation of SAR images difficult. In this letter, we propose a despeckling method by simultaneously exploring low-rank prior and multi-scale prior of SAR images. Especially, we propose a low-rank minimization model by considering a data fidelity term derived from the Fisher–Tippett distribution and a weighted nuclear norm regularization term. Furthermore, we explore the multi-scale prior by selecting similar patches from different scales of the SAR image. The resulting optimization problem is solved by the alternating direction method of multipliers (ADMM). Experiments conducted on both simulated and real SAR images demonstrate that the proposed method can provide promising despeckling results in terms of speckle reduction and texture and edge details preservation.

Plug-and-play inertial forward–backward algorithm for Poisson image deconvolution

Poisson image deconvolution remains an ill-posed research problem consisting of a nonquadratic data-fidelity term and an implicit regularization function. Recently, the plug-and-play (PnP) framework has provided a new method to reformulate the regularizer model to incorporate efficient denoisers. We develop a PnP inertial forward–backward approach (PnP_IFB) for Poisson noise reconstruction. The key advantages over the conventional alternating direction method of multipliers (ADMM)-based scheme are circumventing the matrix inversion operation and requiring less parameter tuning. The scheme can achieve a numerical convergence rate that is comparable to other acceleration strategies such as the fast iterative shrinkage-thresholding algorithm, but it does not depress the reconstruction quality. Moreover, this characteristic acceleration remains efficient when dealing with some nonconvex regularizers. Then, we demonstrate its effectiveness against other state-of-the-art approaches with respect to their deconvolution performance using synthetic images, and the experimental results prove the superiority of the method when using appropriate denoisers.

On Bayesian estimation and proximity operators

There are two major routes to address inverse problems in signal and image processing, such as denoising or deblurring. A first route relies on Bayesian modeling, where prior probabilities embody models of both the distribution of the unknown variables and their statistical dependence with respect to the observed data. Estimation typically relies on the minimization of an expected loss (e.g. minimum mean squared error, or MMSE). The second route designs (often convex) optimization problems involving the sum of a data fidelity term and a penalty term promoting certain types of unknowns (e.g., sparsity, using an ℓ 1 penalty). Well known relations between these two approaches have led to some widely spread misconceptions. In particular, while the so-called Maximum A Posteriori (MAP) estimate with a Gaussian noise model does lead to an optimization problem with a quadratic data-fidelity term, we disprove through explicit examples the common belief that the converse would be true. It was shown [7] , [9] that for denoising in the presence of additive Gaussian noise, for any prior probability on the unknowns, MMSE estimation can be expressed as a penalized least squares problem, with the apparent characteristics of a MAP estimate with Gaussian noise and a (generally) different prior on the unknowns. In other words, the second route is rich enough to build all possible MMSE estimators associated to additive Gaussian noise via a well chosen penalty. We generalize these results beyond Gaussian denoising and characterize noise models for which the same phenomenon occurs. In particular, we prove that with (a variant of) Poisson noise and any prior probability on the unknowns, MMSE estimation can again be expressed as the solution of a penalized least squares optimization problem. For additive scalar denoising the phenomenon holds if and only if the noise distribution is log-concave. In particular, Laplacian denoising can (perhaps surprisingly) be expressed as the solution of a penalized least squares problem. In the multivariate case, the same phenomenon occurs when the noise model belongs to a particular subset of the exponential family. For multivariate additive denoising, the phenomenon holds if and only if the noise is white and Gaussian.

Bilateral filter based total variation regularization for sparse hyperspectral image unmixing

Spectral unmixing of hyperspectral images aims to find the proportion of constituent materials in mixed pixels. The total variation (TV) regularization is widely included in classical sparse regression formulations to exploit the spatial information in hyperspectral data. It promotes piecewise constant transitions in the fractional abundance of the same endmember among neighboring pixels. The TV regularization term, however, usually brings some staircase effects. To alleviate this drawback, we propose a bilateral filter based TV regularization for hyperspectral image unmixing. Then we present an unmixing model that combines a data-fidelity term, a sparsity regularization term, and the new regularization term. To solve the proposed model, we design an algorithm called sparse unmixing via variable splitting augmented Lagrangian and bilateral filter based TV (SUnSAL-BF-TV), under the alternating direction method of multipliers (ADMM) framework. Our experimental results show that our algorithm is effective to unmix both simulated and real hyperspectral data sets.

Euler’s elastica-based algorithm for Parallel MRI reconstruction using SENSitivity Encoding

SENSitivity Encoding (SENSE) is an effective mathematical formulation for reconstructing under-sampled MRI data obtained in Parallel Magnetic Resonance Imaging (Parallel MRI). The functional model includes a regularization term and a data fidelity term which need to be minimized to obtain a high quality MRI result. The proper choice of regularization is essential for image quality. In this paper, we show that Euler’s elastica is an effective regularization for Parallel MRI data reconstruction, and has advantages over the Total Variation (TV) regularization in improving image signal to noise ratio and image relative error. The Euler’s elastica functional is however complex to minimize as it is non-convex, non-smooth, and highly nonlinear. In this paper, we propose a new numerical method to solve Euler’s elastica regularized SENSE efficiently. This algorithm is based on a variable splitting approach and proper relaxation of the functional. Numerical examples are presented to show the effectiveness of the proposed Euler’s elastica algorithm in comparison to Bregman Operator Splitting with Variable Stepsize, a TV based algorithm.

A fast regularized iterative algorithm for fan-beam CT reconstruction

We propose a fast iterative image reconstruction algorithm for normal, short-scan, and super-short-scan fan-beam computed tomography (CT), which aims at iterative reconstruction for low-dose and few-view CT by minimizing a data-fidelity term regularized with a total variation (TV) penalty. The derivation of the algorithm can be outlined as follows. First, the original minimization problem is formulated into a saddle-point (primal-dual) problem by using the Lagrangian duality, to which we apply the alternating projection proximal (APP) algorithm, which belongs to a class of first-order primal-dual methods. Second, we precondition the iterative formula using the modified ramp filter of the filtered back-projection (FBP) reconstruction algorithm in such a way that the solution to this preconditioned iteration perfectly coincides with the solution to the original problem. The resulting algorithm converges quickly to the minimizer of the cost function. To demonstrate the advantages of our method, we perform reconstruction experiments using projection data from both numerical phantoms and real CT data. Both qualitative and quantitative results are presented.

An α -Divergence-Based Approach for Robust Dictionary Learning.

In this paper, a robust sequential dictionary learning (DL) algorithm is presented. The proposed algorithm is motivated from the maximum likelihood perspective on dictionary learning and its link to the minimization of the Kullback-Leibler divergence. It is obtained by using a robust loss function in the data fidelity term of the DL objective instead of the usual quadratic loss. The proposed robust loss function is derived from the α -divergence as an alternative to the Kullback-Leibler divergence, which leads to a quadratic loss. Compared to other robust approaches, the proposed loss has the advantage of belonging to class of redescending M-estimators, guaranteeing inference stability from large deviations from the Gaussian nominal noise model. The algorithm is obtained by solving a sequence of penalized rank-1 matrix approximation problems, where the l1 -norm is introduced as a penalty promoting sparsity and then using a block coordinate descent approach to estimate the unknowns. Performance comparison with similar robust DL algorithms on digit recognition, background removal, and gray-scale image denoising is performed highlighting efficacy of the proposed algorithm.

Robust 1-bit compressed sensing via hinge loss minimization

Abstract This work theoretically studies the problem of estimating a structured high-dimensional signal $\boldsymbol{x}_0 \in{\mathbb{R}}^n$ from noisy $1$-bit Gaussian measurements. Our recovery approach is based on a simple convex program which uses the hinge loss function as data fidelity term. While such a risk minimization strategy is very natural to learn binary output models, such as in classification, its capacity to estimate a specific signal vector is largely unexplored. A major difficulty is that the hinge loss is just piecewise linear, so that its ‘curvature energy’ is concentrated in a single point. This is substantially different from other popular loss functions considered in signal estimation, e.g. the square or logistic loss, which are at least locally strongly convex. It is therefore somewhat unexpected that we can still prove very similar types of recovery guarantees for the hinge loss estimator, even in the presence of strong noise. More specifically, our non-asymptotic error bounds show that stable and robust reconstruction of $\boldsymbol{x}_0$ can be achieved with the optimal oversampling rate $O(m^{-1/2})$ in terms of the number of measurements $m$. Moreover, we permit a wide class of structural assumptions on the ground truth signal, in the sense that $\boldsymbol{x}_0$ can belong to an arbitrary bounded convex set $K \subset{\mathbb{R}}^n$. The proofs of our main results rely on some recent advances in statistical learning theory due to Mendelson. In particular, we invoke an adapted version of Mendelson’s small ball method that allows us to establish a quadratic lower bound on the error of the first-order Taylor approximation of the empirical hinge loss function.

Local-linear-fitting-based matting for joint hole filling and depth upsampling of RGB-D images

We propose an approach for jointly filling holes and upsampling depth information for RGB-D images captured with common acquisition systems, where RGB color information is available at all pixel locations whereas depth information is only available at lower resolution and entirely missing in small regions referred to as “holes.” Depth information completion is formulated as a minimization of an objective function composed of two additive terms. The first data fidelity term penalizes disagreement with the observed low-resolution data. The second regularization term penalizes weighted depth deviations from a local linear model in spatial coordinates, where the weights are experimentally determined to ensure consistency between the RGB color image and the estimated depth image. Analogous to techniques used for optimization formulations of image matting, the completed depth image is then obtained by solving a large sparse linear system of equations. We also propose a memory-efficient implementation of the proposed method based on the conjugate gradient method. Visual evaluation of results obtained with the proposed algorithm demonstrates that the method provides high-resolution depth maps that are consistent with the color images. Furthermore, the memory-efficient implementation significantly reduces memory requirements, allowing for computation of the upsampled, hole-filled depth maps for typical RGB-D images on normal workstation hardware. Quantitative comparisons demonstrate that the method offers an improvement in accuracy over the current state-of-the-art techniques for depth information completion. Importantly, statistical analysis, which we present in this paper, also reveals that prior evaluations of depth upsampling accuracy are potentially biased because the evaluations inappropriately used preprocessed hole-filled data as “ground truth.” An implementation of the proposed algorithm can be accessed and executed through Code Ocean: https://codeocean.com/capsule/5103691/tree/v1.

SAR Image Despeckling Based on Nonlocal Low-Rank Regularization

In this paper, we propose a new synthetic aperture radar (SAR) image despeckling method based on the nonlocal low-rank minimization model. First, some similar image patches are selected for each pixel to construct the patch group matrix (PGM). Then, a new low-rank minimization model, called Fisher–Tippett distribution (FT)-weighted nuclear norm minimization (WNNM), is proposed to recover the underlying low-rank component from the PGM. Specifically, the FT-WNNM is developed by reformulating the despeckling problem as the maximizing a posterior probability problem. The new model consists of a data fidelity term and a regularization term (also called prior term). The data fidelity term is derived from the statistical distribution of SAR images in the logarithm domain, which is known as the Fisher–Tippett distribution, and the regularization term is the recent weighted nuclear norm. Then, the alternating direction method of multipliers (ADMM) is introduced to solve the corresponding optimization problem. Under ADMM framework, the resulting subproblems can be solved efficiently and the convergence can be guaranteed. Extensive experiments on both simulated and real SAR images demonstrate that the proposed method can achieve comparable or even better despeckling performance than some state-of-the-art despeckling algorithms.

-minimization methods for image restoration problems based on wavelet frames

In this paper we consider a class of -minimization and wavelet frame-based models for image deblurring and denoising. Mathematically, they can be formulated as minimizing the sum of a data fidelity term and the l0-‘norm’ of the framelet coefficients of the underlying image, and we are particularly interested in three different types of data fidelity forms for image restoration problems. We first study the first-order optimality conditions for these models. We then propose a penalty decomposition (PD) method for solving these problems in which a sequence of penalty subproblems are solved by a block coordinate descent (BCD) method. Under some suitable assumptions, we establish that any accumulation point of the sequence generated by the PD method satisfies the first-order optimality conditions of these problems. Moreover, for the problems in which the data fidelity term is convex, we show that such an accumulation point is a local minimizer of the problems. In addition, we show that any accumulation point of the sequence generated by the BCD method is a block coordinate minimizer of the penalty subproblem. Furthermore, under some convexity assumptions on the data fidelity term, we prove that such an accumulation point is a local minimizer of the penalty subproblem. Numerical simulations show that the proposed -minimization methods enjoy great potential for image deblurring and denoising in terms of solution quality and/or speed.

A novel medical image fusion by combining TV-L1 decomposed textures based on adaptive weighting scheme

Medical image fusion involves combining multiple images of diverse modalities to acquire superior image quality while retaining the information of an individual image. The fusion process aids physicians to diagnose and assess the disease by increasing the visual information and clarity. Direct fusion methods often generate undesirable effects leading to distortion and poor contrast. To some extent, multi-scale decomposition (MSD) methods have achieved success in various image fusion problems. However, they suffer from ringing artifacts, due to the blur caused by strong edges in decomposition steps. Hence, to overcome these drawbacks, we propose an efficient and novel image fusion algorithm for constructing a fused image through total variation (TV-L1) model using an optimized adaptive weighting scheme. Our method focuses on transferring the prominent textural details of Magnetic Resonance Imaging (MRI) and visual details of Positron Emission Tomography (PET) by cartoon and texture decomposition. It consists of data fidelity term to force the cartoon component to remain close to original image and a regularization term to transfer gradients from original image to the fused image. The present work preserves structural and functional details with high contrast. Experiments were conducted using MRI and PET images collected from standard Brain Atlas Datasets. The qualitative and quantitative analysis of the work suggest that the proposed method achieves better quality and accuracy as compared with the available state-of-the-art techniques.

Subspace Clustering Under Complex Noise

In this paper, we study the subspace clustering problem under complex noise. A wide class of reconstruction-based methods model the subspace clustering problem by combining a quadratic data-fidelity term and a regularization term. In a statistical framework, the data-fidelity term assumes to be contaminated by a unimodal Gaussian noise, which is a popular setting in most current subspace clustering models. However, the realistic noise is much more complex than our assumptions. Besides, the coarse representation of the data-fidelity term may depress the clustering accuracy, which is often used to evaluate the models. To address this issue, we propose the mixture of Gaussian regression (MoG Regression) for subspace clustering. The MoG Regression seeks a valid way to model the unknown noise distribution, which approaches the real one as far as possible, so that the desired affinity matrix is better at characterizing the structure of data in the real world, and furthermore, improving the performance. Theoretically, the proposed model enjoys the grouping effect, which encourages the coefficients of highly correlated points are nearly equal. Drawing upon the ideal of the minimum message length, a model selection strategy is proposed to estimate the numbers of the Gaussian components that shows a way how to seek the number of Gaussian components besides determining it by empirical value. In addition, the asymptotic property of our model is investigated. The proposed model is evaluated on the challenging datasets. The experimental results show that the proposed MoG Regression model significantly outperforms several state-of-the-art subspace clustering methods.

Robust Compressive Two-Dimensional Near-Field Millimeter-Wave Image Reconstruction in Impulsive Noise

In this letter, a novel robust two-dimensional nearfield millimeter-wave (MMW) image reconstruction algorithm is developed based on compressed sensing (CS) in the presence of impulsive measurement noise. To enable the sparse MMW image recovery, the CS algorithm usually employs the popular l 2 -norm as the data fidelity term and a combination of multiple sparsityinduced functions as the penalty term. The l 2 -norm is non-robust against impulsive noise since the Gaussian noise distribution assumption is not valid, and the presence of impulsive noise will severely degrade the robustness of the compressive MMW image recovery. To gain more robustness performance, a complex correntropy based data-fitting term, namely complex correntropic loss (CC-loss), is used to replace the l2-norm for the near-field MMW measurement contaminated by impulsive noise. In order to solve the corresponding minimization problem, an additive half quadratic method is used to transform the CC-loss term to its convex form, and then a parallel primal-dual process is used to guarantee the convergence of the proposed algorithm. In total, 150-GHz MMW experimental results show that the proposed algorithm can achieve accurate image reconstruction from compressive measurements under different impulsive noise levels.

A DICTIONARY LEARNING APPROACH FOR FRACTAL IMAGE CODING

In recent years, sparse representations of images have shown to be efficient approaches for image recovery. Following this idea, this paper investigates incorporating a dictionary learning approach into fractal image coding, which leads to a new model containing three terms: a patch-based sparse representation prior over a learned dictionary, a quadratic term measuring the closeness of the underlying image to a fractal image, and a data-fidelity term capturing the statistics of Gaussian noise. After the dictionary is learned, the resulting optimization problem with fractal coding can be solved effectively. The new method can not only efficiently recover noisy images, but also admirably achieve fractal image noiseless coding/compression. Experimental results suggest that in terms of visual quality, peak-signal-to-noise ratio, structural similarity index and mean absolute error, the proposed method significantly outperforms the state-of-the-art methods.

A Global Method for Non-Rigid Registration of Cell Nuclei in Live Cell Time-Lapse Images.

Non-rigid registration of cell nuclei in time-lapse microscopy images can be achieved through estimating the deformation fields using optical flow methods. In contrast to local optical flow models employed in the existing non-rigid registration methods, we introduce approaches based on a global optical flow model. Our registration model consists of a data fidelity term and a regularization term. We compared different regularizers for the deformation fields and found that a convex quadratic function is more suitable than non-convex ones. To improve the robustness, we propose an adaptive weighting scheme based on the statistics of the noise in fluorescence microscopy images as well as a combined local-global scheme. Moreover, we extend the global method by exploiting high-order image features. The best suitable high-order features are determined through learning two generative image models, namely, fields of experts and convolutional Gaussian restricted Boltzmann machine, whose model formulations are both consistent with the assumption of high-order feature constancy in the registration model. Using multiple data sets of real 2D and 3D live cell microscopy image sequences as well as synthetic image data, we demonstrate that our proposed approach outperforms the previous methods in terms of both registration accuracy and computational efficiency.