Inverse Problems In Imaging Research Articles

Guest Editorial Special Issue on Geometric Deep Learning in Medical Imaging

In recent years, more and more attention has been devoted to geometric deep learning (GDL) and its applications to various problems in medical imaging. Unlike convolutional neural networks (CNNs) limited to 2-D/3-D grid-structured data, GDL can handle non-Euclidean data (i.e., graphs and manifolds) and is hence well-suited for medical imaging data such as structure-function connectivity networks, imaging genetics and omics, spatio-temporal anatomical representations, physics-informed GDL for optimal imaging sampling and acquisition, GDL in imaging inverse problems, etc. However, despite recent advances in GDL research, questions remain on how best to learn representations of non-Euclidean medical imaging data; how to convolve effectively on graphs; how to perform graph pooling/unpooling; how to handle heterogeneous data; and how to improve the interpretability of GDL. After discussing many other domain experts, we identify the need for a special issue that brings to the attention of the medical imaging community these interesting topics.

Using an LSTM network to monitor industrial reactors using electrical capacitance and impedance tomography – a hybrid approach

The article presents a new concept for monitoring industrial tank reactors. The presented concept allows for faster and more reliable monitoring of industrial processes, which increases their reliability and reduces operating costs. The innovative method is based on electrical tomography. At the same time, it is non-invasive and enables the imaging of phase changes inside tanks filled with liquid. In particular, the hybrid tomograph can detect gas bubbles and crystals formed during industrial processes. The main novelty of the described solution is the simultaneous use of two types of electrical tomography: impedance and capacitance. Another novelty is the use of the LSTM network to solve the tomographic inverse problem. It was made possible by taking the measurement vector as a data sequence. Research has shown that the proposed hybrid solution and the LSTM algorithm work better than separate systems based on impedance or capacitance tomography.

VGenNet: Variable Generative Prior Enhanced Single Pixel Imaging

Single-pixel imaging (SPI) is an emerging imaging methodology that converts a two- or even three-dimensional image acquisition problem into a one-dimensional (1D) temporal-signal detection problem. Thus, it is crucially important to develop efficient SPI techniques for image reconstruction from the 1D measurements, in particular, an undersampled one. Recently, various studies have demonstrated the superiority of deep learning for SPI. However, due to the generalization issue, conventional data-driven deep learning is a task-specific approach. One needs to retrain the neural network for different SPI imaging problems and different types of objects. Here, we propose a variable generative network enhanced SPI algorithm (VGenNet) by incorporating a model-driven fine-tuning process into a generative model that may have been trained for other tasks. VGenNet simultaneously updates the input vector and the weights in a generator to generate feasible solutions that reproduce the raw measurements. We demonstrate the proposed technique with indoor SPI and outdoor 3D single-pixel LiDAR experiments. Our results show that high-quality images can be reconstructed at low sampling ratios under different system configurations, demonstrating the good performance and flexibility of VGenNet. Overall, the proposed VGenNet is a general framework to take advantage of both the data and physics priors, allowing the direct use of a pretrained generative model to solve various inverse imaging problems.

Convergence Results for Primal-Dual Algorithms in the Presence of Adjoint Mismatch

Most optimization problems arising in imaging science involve high-dimensional linear operators and their adjoints. In the implementations of these operators, changes may be introduced for various practical considerations (e.g., memory limitation, computational cost, convergence speed), leading to an adjoint mismatch. This occurs for the X-ray tomographic inverse problems found in computed tomography (CT), where a surrogate operator often replaces the adjoint of the measurement operator (called the projector). The resulting adjoint mismatch can jeopardize the convergence properties of iterative schemes used for image recovery. In this paper, we study the theoretical behavior of a panel of primal-dual proximal algorithms, which rely on forward-backward-(forward) splitting schemes when an adjoint mismatch occurs. We analyze these algorithms by focusing on the resolution of possibly nonsmooth convex penalized minimization problems in an infinite-dimensional setting. Using tools from fixed point theory, we show that they can solve monotone inclusions beyond minimization problems. Such findings indicate that these algorithms can be seen as a generalization of classical primal-dual formulations. The applicability of our findings is also demonstrated through two numerical experiments in the context of CT image reconstruction.

Posterior sampling for inverse imaging problems on the sphere in seismology and cosmology

ABSTRACT In this work, we describe a framework for solving spherical inverse imaging problems using posterior sampling for full uncertainty quantification. Inverse imaging problems defined on the sphere arise in many fields, including seismology and cosmology, where images are defined on the globe and the cosmic sphere, and are generally high-dimensional and computationally expensive. As a result, sampling the posterior distribution of spherical imaging problems is a challenging task. Our framework leverages a proximal Markov chain Monte Carlo (MCMC) algorithm to efficiently sample the high-dimensional space of spherical images with a sparsity-promoting wavelet prior. We detail the modifications needed for the algorithm to be applied to spherical problems, and give special consideration to the crucial forward modelling step, which contains computationally expensive spherical harmonic transforms. By sampling the posterior, our framework allows for full and flexible uncertainty quantification, something which is not possible with other methods based on, for example, convex optimization. We demonstrate our framework in practice on full-sky cosmological mass-mapping and to the construction of phase velocity maps in global seismic tomography. We find that our approach is potentially useful at moderate resolutions, such as those of interest in seismology. However, at high resolutions, such as those required for astrophysical applications, the poor scaling of the complexity of spherical harmonic transforms severely limits our method, which may be resolved with future GPU implementations. A new Python package, pxmcmc, containing the proximal MCMC sampler, measurement operators, wavelet transforms, and sparse priors are made publicly available.

Interpretable Deep Attention Prior for Image Restoration and Enhancement

An inductive bias induced by an untrained network architecture has been shown to be effective as a deep image prior (DIP) in solving inverse imaging problems. However, it is still unclear as to what kind of prior is encoded in the network architecture, and the early stopping for the overfitting problem of DIP still remains the challenge. To address this, we introduce an interpretable network that explores self-attention as a deep attention prior (DAP). Specifically, the proposed deep attention prior is formulated as an interpretable optimization problem. A nonlocal self-similarity prior is incorporated into the network architecture by a self-attention mechanism. Each attention map from our proposed DAP reveals how an output value is generated, which leads to a better understanding of the prior. Furthermore, compared to DIP, the proposed DAP regards the single input degraded image as input to reduce the instability, and introduces the mask operation to handle the early stopping problem. Experiments show that the proposed network works as an effective image prior for solving different inverse imaging problems, such as denoising, inpainting, or pansharpening, while also showing potential applications in higher-level processing such as interactive segmentation and selective colorization.

Cycle-Consistency-Based Uncertainty Quantification of Neural Networks in Inverse Imaging Problems

Uncertainty estimation is critical for numerous deep neural network (DNN) applications and has drawn increasing attention from researchers. In this study, we demonstrated an uncertainty quantification approach for DNNs used in inverse problems based on cycle consistency. We built forward–backward cycles using the available physical forward model and a trained DNN solving the inverse problem at hand and accordingly derived uncertainty estimators through regression analysis on the consistency of these forward–backward cycles. We theoretically analyzed the cycle consistency metrics and derived their relationship with the uncertainty, bias, and robustness of neural network inference. To demonstrate the effectiveness of these cycle-consistency-based uncertainty estimators, we classified corrupted and out-of-distribution input image data using widely used image deblurring and super-resolution neural networks as test beds. Our blind tests demonstrated that our method surpassed other models in detecting previously unseen data corruption and distribution shifts. This study provides a simple-to-implement and rapid uncertainty quantification method that can be universally applied to various neural networks used to solve inverse problems.

SLN-RED: Regularization by Simultaneous Local and Nonlocal Denoising for Image Restoration

Regularization by denoising (RED) framework has shown impressive performance for many imaging inverse problems, by leveraging the denoising method in defining an explicit regularization. In this letter, we propose a novel SLN-RED scheme for image restoration by exploiting the local and nonlocal denoisers simultaneously. Theoretically, we proves that for <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">bounded</i> denoisers, the SLN-RED under ADMM scheme with a continuation strategy converges to a fixed-point. Numerical experiments on deblurring and super-resolution tasks demonstrate promising performance of the proposed algorithm.

Self-Supervised Coordinate Projection Network for Sparse-View Computed Tomography

Sparse-view Computed Tomography (SVCT) has great potential for decreasing patient radiation exposure dose during scanning. In this work, we propose a Self-supervised COordinate Projection nEtwork (SCOPE) to reconstruct the artifact-free CT image from the acquired SV sinogram by solving the inverse problem of tomography imaging. To solve the under-determined inverse imaging problem, we first introduce an implicit neural representation (INR) network to constrain the solution space via image continuity prior. And inspired by the relationship between linear algebra and inverse problems, we propose a novel re-projection strategy to generate a dense view sinogram from the initial solution, which significantly improves the rank of the linear equation system and produces a more stable CT image solution space. Specially, the desired CT image is represented as an implicit function of the two-dimensional spatial coordinate to directly approximate the SV sinogram through the CT imaging forward model. Then, a dense-view sinogram is generated from the fine-trained INR network. The final CT reconstruction is reconstructed by applying Filtered Back Projection (FBP) to the generated dense-view sinogram. Additionally, we integrate the recent hash encoding into our SCOPE model, which efficiently accelerates the model training process. We evaluate SCOPE in parallel and fan X-ray beam SVCT reconstruction tasks. Our experiment results demonstrate that the re-projection strategy significantly improves the image reconstruction quality (+3 dB for PSNR at least). The proposed SCOPE model provides state-of-the-art reconstruction results compared to two latest INR-based methods and two well-popular supervised DL methods for the SV CT image reconstruction.

Generative Models for Inverse Imaging Problems: From mathematical foundations to physics-driven applications

Physics-informed generative modeling for inverse problems in computational imaging is a fast-growing field encompassing a variety of methods and applications. Here, we review a few generative modeling techniques, such as variational autoencoders (VAEs) and generative adversarial networks (GANs), as well as more recent developments in score-based generative models. Through different imaging applications, we review how the generative modeling techniques are effectively combined with the physics of the imaging problem, e.g., the measurement forward model and physical properties of the target objects, to solve the inverse problems.

Deep Equilibrium Learning of Explicit Regularization Functionals for Imaging Inverse Problems

Deep Equilibrium Learning of Explicit Regularization Functionals for Imaging Inverse Problems

On Maximum a Posteriori Estimation with Plug &amp; Play Priors and Stochastic Gradient Descent

Bayesian methods to solve imaging inverse problems usually combine an explicit data likelihood function with a prior distribution that explicitly models expected properties of the solution. Many kinds of priors have been explored in the literature, from simple ones expressing local properties to more involved ones exploiting image redundancy at a non-local scale. In a departure from explicit modelling, several recent works have proposed and studied the use of implicit priors defined by an image denoising algorithm. This approach, commonly known as Plug & Play (PnP) regularization, can deliver remarkably accurate results, particularly when combined with state-of-the-art denoisers based on convolutional neural networks. However, the theoretical analysis of PnP Bayesian models and algorithms is difficult and works on the topic often rely on unrealistic assumptions on the properties of the image denoiser. This papers studies maximum a posteriori (MAP) estimation for Bayesian models with PnP priors. We first consider questions related to existence, stability and well-posedness and then present a convergence proof for MAP computation by PnP stochastic gradient descent (PnP-SGD) under realistic assumptions on the denoiser used. We report a range of imaging experiments demonstrating PnP-SGD as well as comparisons with other PnP schemes.

Guided Deep Generative Model-Based Spatial Regularization for Multiband Imaging Inverse Problems.

When adopting a model-based formulation, solving inverse problems encountered in multiband imaging requires to define spatial and spectral regularizations. In most of the works of the literature, spectral information is extracted from the observations directly to derive data-driven spectral priors. Conversely, the choice of the spatial regularization often boils down to the use of conventional penalizations (e.g., total variation) promoting expected features of the reconstructed image (e.g., piece-wise constant). In this work, we propose a generic framework able to capitalize on an auxiliary acquisition of high spatial resolution to derive tailored data-driven spatial regularizations. This approach leverages on the ability of deep learning to extract high level features. More precisely, the regularization is conceived as a deep generative network able to encode spatial semantic features contained in this auxiliary image of high spatial resolution. To illustrate the versatility of this approach, it is instantiated to conduct two particular tasks, namely multiband image fusion and multiband image inpainting. Experimental results obtained on these two tasks demonstrate the benefit of this class of informed regularizations when compared to more conventional ones.

Regularization by Denoising for Inverse Problems in Imaging

In this work, a generalized scheme of regularization of inverse problems is considered, where a priori knowledge about the smoothness of the solution is given by means of some self-adjoint operator in the solution space. The formulation of the problem is considered, namely, in addition to the main inverse problem, an additional problem is defined, in which the solution is the right-hand side of the equation. Thus, for the regularization of the main inverse problem, an additional inverse problem is used, which brings information about the smoothness of the solution to the initial problem. This formulation of the problem makes it possible to use operators of high complexity for regularization of inverse problems, which is an urgent need in modern machine learning problems, in particular, in image processing problems. The paper examines the approximation error of the solution of the initial problem using an additional problem.

Theoretical and numerical comparison of first order algorithms for cocoercive equations and smooth convex optimization

This paper provides a theoretical and numerical comparison of classical first-order splitting methods for solving smooth convex optimization problems and cocoercive equations. From a theoretical point of view, we compare convergence rates of gradient descent, forward-backward, Peaceman-Rachford, and Douglas-Rachford algorithms for minimizing the sum of two smooth convex functions when one of them is strongly convex. A similar comparison is given in the more general cocoercive setting under the presence of strong monotonicity and we observe that the convergence rates in optimization are strictly better than the corresponding rates for cocoercive equations for some algorithms. We obtain improved rates with respect to the literature in several instances by exploiting the structure of our problems. Moreover, we indicate which algorithm has the smallest convergence rate depending on strong convexity and cocoercive parameters. From a numerical point of view, we verify our theoretical results by implementing and comparing previous algorithms in well established signal and image inverse problems involving sparsity. We replace the widely used ℓ1 norm with the Huber loss and we observe that fully proximal-based strategies have numerical and theoretical advantages with respect to methods using gradient steps.

Active mode single-pixel non-line-of-sight imaging system based on second-order correlation and diffraction

Non-line-of-sight (NLOS) imaging relies on capturing light that has been weakened due to multiple reflections and recovering the occluded scene with high quality is very challenging. To improve the quality of NLOS imaging, an active mode single-pixel NLOS imaging reconstruction framework based on the second-order correlation function and diffraction inverse operation is proposed. Under the collective effect of illumination pattern and single-pixel detector light intensity, the inverse imaging problem is first solved by a statistically weighted average, and then, an additional process is connected to reduce the influence of diffraction on reconstructed image quality based on the Fourier transform and inverse diffraction operation. The simulation and experimental results show that the proposed method with the Hadamard pattern and diffraction inverse operation, which achieved the SSIM value of 0.9286, is superior to the random pattern at a full sampling rate. The lens aperture size, pattern location, and the number of measurements have a significant effect on the system. Therefore, the proposed scheme has potential practical applications.

One-to-all lightweight Fourier channel attention convolutional neural network for speckle reconstructions.

Speckle reconstruction is a classical inverse problem in computational imaging. Inspired by the memory effect of the scattering medium, deep learning methods reveal excellent performance in extracting the correlation of speckle patterns. Nowadays, advanced models generally include more than 10M parameters and mostly pay more attention to the spatial feature information. However, the frequency domain of images also contains precise hierarchical representations. Here we propose a one-to-all lightweight Fourier channel attention convolutional neural network (FCACNN) with Fourier channel attention and the res-connected bottleneck structure. Compared with the state-of-the-art model, i.e., self-attention armed convolutional neural network (SACNN), our architecture has better feature extraction and reconstruction ability. The Pearson correlation coefficient and Jaccard index scores of FCACNN increased by at least 5.2% and 13.6% compared with task-related models. And the parameter number of the lightweight FCACNN is only 1.15M. Furthermore, the validation results show that the one-to-all model, FCACNN, has excellent generalization capability on unseen speckle patterns such as handwritten letters and Quickdraws.

Solving the inverse problem in electrocardiography imaging for atrial fibrillation using various time-frequency decomposition techniques based on empirical mode decomposition: A comparative study.

Electrocardiographic imaging (ECGI) can aid in identifying the driving sources that cause and sustain atrial fibrillation (AF). Traditional regularization strategies for addressing the ECGI inverse problem are not currently concerned about the multi-scale analysis of the inverse problem, and these techniques are not clinically reliable. We have previously investigated the solution based on uniform phase mode decomposition (UPEMD-based) to the ECGI inverse problem. Numerous other methods for the time-frequency analysis derived from empirical mode decomposition (EMD-based) have not been applied to the inverse problem in ECGI. By applying many EMD-based solutions to the ECGI inverse problem and evaluating the performance of these solutions, we hope to find a more efficient EMD-based solution to the ECGI inverse problem. In this study, five AF simulation datasets and two real datasets from AF patients derived from a clinical ablation procedure are employed to evaluate the operating efficiency of several EMD-based solutions. The Pearson's correlation coefficient (CC), the relative difference measurement star (RDMS) of the computed epicardial dominant frequency (DF) map and driver probability (DP) map, and the distance (Dis) between the estimated and referenced most probable driving sources are used to evaluate the application of various EMD-based solutions in ECGI. The results show that for DF maps on all simulation datasets, the CC of UPEMD-based and improved UPEMD (IUPEMD)-based techniques are both greater than 0.95 and the CC of the empirical wavelet transform (EWT)-based solution is greater than 0.889, and the RDMS of UPEMD-based and IUPEMD-based approaches is less than 0.3 overall and the RDMS of EWT-based method is less than 0.48, performing better than other EMD-based solutions; for DP maps, the CC of UPEMD-based and IUPEMD-based techniques are close to 0.5, the CC of EWT-based is 0.449, and the CC of the remaining EMD-based techniques on the SAF and CAF is all below 0.1; the RDMS of UPEMD-based and IUPEMD-based are 0.06∼0.9 less than that of other EMD-based methods for all the simulation datasets overall. On two authentic AF datasets, the Dis between the first 10 real and estimated maximum DF positions of UPEMD-based and EWT-based methods are 212∼1440 less than that of others, demonstrating these two EMD-based solutions are superior and are suggested for clinical application in solving the ECGI inverse problem. On all datasets, EWT-based algorithms deconstruct the signal in the shortest time (no more than 0.12s), followed by UPEMD-based solutions (less than 0.81s), showing that these two schemes are more efficient than others.

SIGEST

The SIGEST article in this issue is “A Proximal Markov Chain Monte Carlo Method for Bayesian Inference in Imaging Inverse Problems: When Langevin Meets Moreau,” by Alain Durmus, Éric Moulines, and Marcelo Pereyra. The authors provide new algorithms to sample from high-dimensional log-concave probability measures, where they combine Moreau--Yosida envelopes with the Euler--Maruyama discretization of Langevin diffusions. This allows for an efficient Markov chain Monte Carlo methodology that is applicable to inverse problems arising in imaging sciences. Asymptotic and nonasymptotic convergence results are provided, along with extensive computational experiments on realistic imaging problems involving deconvolution and tomographic reconstruction. The original article, which appeared in the SIAM Journal on Imaging Sciences in 2018, has attracted substantial interest. In preparing this highlighted SIGEST version, the authors have expanded the introduction to make it accessible to a wide audience. The final section also discusses follow-up work arising from the original publication.

CNN-Based Deep Learning Architecture for Electromagnetic Imaging of Rough Surface Profiles

A convolutional neural network (CNN)-based deep learning (DL) technique for electromagnetic (EM) imaging of rough surfaces separating two dielectric media is presented. The direct scattering problem is formulated through the conventional integral equations, and the synthetic scattered field data are produced by a fast numerical solution technique, which is based on method of moments (MoM). Two different special CNN architectures are designed and implemented for the solution of the inverse rough surface imaging problem, wherein both random and deterministic rough surface profiles can be imaged. It is shown by a comprehensive numerical analysis that the proposed DL inversion scheme is very effective and robust.