Inverse Problems In Imaging Research Articles

CTDNet: cartoon-texture decomposition-based gray image super-resolution network with multiple degradations

In the case of multiple degradations, current deep-learning-based gray image super-resolution (SR) methods equally process all components in an image, resulting in missing subtle details. To address this issue, we elaborate a cartoon-texture decomposition-based (CTD) module that can automatically decompose an image into a smooth cartoon component and an oscillatory texture component. The CTD module is a plug-and-play prior module that can be applied in solving imaging inverse problems. Specifically, for the SR task under multiple degradations, we apply CTD as a prior module to build an unfolding SR network termed CTDNet. For the SR task of real terahertz images, the boundary (i.e., the boundary between the object of interest and the carrier table) recovered by CTDNet has artifacts, which limits its realistic applications. To reduce these boundary artifacts, we post-process the SR terahertz images by using a boundary artifact reduction method. Experimental results on the synthetic dataset and real terahertz images demonstrate that the proposed algorithms can maintain subtle details and achieve comparable visual results. The code can be found at https://github.com/shibaoshun/CTDNet.

A layer potential approach to inverse problems in brain imaging

Abstract We study the inverse source localisation problem using the electric potential measured point-wise inside the head with stereo-ElectroEncephaloGraphy (sEEG), the electric potential measured point-wise on the scalp with ElectroEncephaloGraphy (EEG) or the magnetic flux density measured point-wise outside the head with MagnetoEncephaloGraphy (MEG). We present a method that works on a wide range of models of primary currents; in particular, we give details for primary currents that are assumed to be smooth vector fields that are supported on and normally oriented to the grey/white matter interface. Irrespective of the data used, we also solve the transmission problem of the electric potential associated with a recovered source; hence we solve the cortical mapping problem. To ensure that the electric potential and normal currents are continuous in the head, the electric potential is expressed as a linear combination of double layer potentials and the magnetic flux density is expressed as a linear combination of single layer potentials. Numerically, we solve the problems on meshed surfaces of the grey/white matter interface, cortical surface, skull and scalp. A main feature of the numerical approach we take is that, on the meshed surfaces, we can compute the double and single layer potentials exactly and at arbitrary points. Because we explicitly study the transmission of the electric potential in head when using magnetic data, the coupling of electric and magnetic data in the source recovery problem is made explicit and shows the advantage of using simultaneous electric and magnetic data. We provide numerical examples of the source recovery and inverse cortical mapping using synthetic data.

MMV-Net: A Multiple Measurement Vector Network for Multifrequency Electrical Impedance Tomography.

Multifrequency electrical impedance tomography (mfEIT) is an emerging biomedical imaging modality to reveal frequency-dependent conductivity distributions in biomedical applications. Conventional model-based image reconstruction methods suffer from low spatial resolution, unconstrained frequency correlation, and high computational cost. Deep learning has been extensively applied in solving the EIT inverse problem in biomedical and industrial process imaging. However, most existing learning-based approaches deal with the single-frequency setup, which is inefficient and ineffective when extended to the multifrequency setup. This article presents a multiple measurement vector (MMV) model-based learning algorithm named MMV-Net to solve the mfEIT image reconstruction problem. MMV-Net considers the correlations between mfEIT images and unfolds the update steps of the Alternating Direction Method of Multipliers for the MMV problem (MMV-ADMM). The nonlinear shrinkage operator associated with the weighted l2,1 regularization term of MMV-ADMM is generalized in MMV-Net with a cascade of a Spatial Self-Attention module and a Convolutional Long Short-Term Memory (ConvLSTM) module to better capture intrafrequency and interfrequency dependencies. The proposed MMV-Net was validated on our Edinburgh mfEIT Dataset and a series of comprehensive experiments. The results show superior image quality, convergence performance, noise robustness, and computational efficiency against the conventional MMV-ADMM and the state-of-the-art deep learning methods.

Regularising Inverse Problems with Generative Machine Learning Models

Deep neural network approaches to inverse imaging problems have produced impressive results in the last few years. In this survey paper, we consider the use of generative models in a variational regularisation approach to inverse problems. The considered regularisers penalise images that are far from the range of a generative model that has learned to produce images similar to a training dataset. We name this family generative regularisers. The success of generative regularisers depends on the quality of the generative model and so we propose a set of desired criteria to assess generative models and guide future research. In our numerical experiments, we evaluate three common generative models, autoencoders, variational autoencoders and generative adversarial networks, against our desired criteria. We also test three different generative regularisers on the inverse problems of deblurring, deconvolution, and tomography. We show that restricting solutions of the inverse problem to lie exactly in the range of a generative model can give good results but that allowing small deviations from the range of the generator produces more consistent results. Finally, we discuss future directions and open problems in the field.

Image restoration via joint low-rank and external nonlocal self-similarity prior

Recent studies have revealed that joint priors, such as joint sparsity and external nonlocal self-similarity (ENSS) prior and joint low-rank and sparsity prior, are extremely effective in various image inverse problems. Few works, however, make use of both low-rank and ENSS priors. With this in mind, in this paper we propose a new joint prior, namely LRENSS prior, which utilizes low-rank and ENSS priors jointly in a unified framework, and successfully adapt the proposed LRENSS prior to image restoration problems. Specifically, low-rank and ENSS priors are bridged by treating ENSS prior as dictionaries for structural sparse representation. Further, an elegant block coordinate descent method is developed to solve the corresponding optimization problem. The proposed LRENSS prior is validated on image denoising and image deblurring tasks. Experimental results illustrate that the proposed LRENSS prior has better performance than other state-of-the-art algorithms in both qualitative and quantitative assessments.

Learning to sound imaging by a model-based interpretable network

Acoustic beamforming methods based on microphone arrays have been widely used for sound source localization in various industrial fields. The conventional methods such as Delay and Sum (DAS) beamforming are limited by poor spatial resolution although their computational complexity is low. Efforts have been made to improve the resolution of the beamforming map, achieved by both model-based and deep network-based approaches. However, the model-based methods usually suffer from high additional computational effort and largely rely on selecting user-determined parameters. The deep network-based methods may make it difficult to guarantee generalization as well as hard to generate the beamforming map directly. In this paper, we first propose a scheme DAMAS-FISTA-LASSO to solve the inverse problem of sound imaging, significantly reducing runtime requirements. By deep unfolding the proposed algorithm, we further have designed an end-to-end interpretable model-based deep neural network termed DAMAS-FISTA-LASSO-Net (DFLNet) for real-time and high-resolution mapping of acoustic sources, which could obtain satisfying performance on both simulated and real-world data, demonstrating the strong generalization capability of the network.

Uncertainty quantification of the effect of cardiac position variability in the inverse problem of electrocardiographic imaging

Objective. Electrocardiographic imaging (ECGI) is a functional imaging modality that consists of two related problems, the forward problem of reconstructing body surface electrical signals given cardiac bioelectric activity, and the inverse problem of reconstructing cardiac bioelectric activity given measured body surface signals. ECGI relies on a model for how the heart generates bioelectric signals which is subject to variability in inputs. The study of how uncertainty in model inputs affects the model output is known as uncertainty quantification (UQ). This study establishes develops, and characterizes the application of UQ to ECGI. Approach. We establish two formulations for applying UQ to ECGI: a polynomial chaos expansion (PCE) based parametric UQ formulation (PCE-UQ formulation), and a novel UQ-aware inverse formulation which leverages our previously established ‘joint-inverse’ formulation (UQ joint-inverse formulation). We apply these to evaluate the effect of uncertainty in the heart position on the ECGI solutions across a range of ECGI datasets. Main results. We demonstrated the ability of our UQ-ECGI formulations to characterize the effect of parameter uncertainty on the ECGI inverse problem. We found that while the PCE-UQ inverse solution provided more complex outputs such as sensitivities and standard deviation, the UQ joint-inverse solution provided a more interpretable output in the form of a single ECGI solution. We find that between these two methods we are able to assess a wide range of effects that heart position variability has on the ECGI solution. Significance. This study, for the first time, characterizes in detail the application of UQ to the ECGI inverse problem. We demonstrated how UQ can provide insight into the behavior of ECGI using variability in cardiac position as a test case. This study lays the groundwork for future development of UQ-ECGI studies, as well as future development of ECGI formulations which are robust to input parameter variability.

Bilevel inverse problems in neuromorphic imaging

Event or neuromorphic cameras are novel biologically inspired sensors that record data based on the change in light intensity at each pixel asynchronously. They have a temporal resolution of microseconds. This is useful for scenes with fast moving objects that can cause motion blur in traditional cameras, which record the average light intensity over an exposure time for each pixel synchronously. This paper presents a bilevel inverse problem framework for neuromorphic imaging. Existence of solution to the inverse problem is established. Second order sufficient conditions are derived under special situations for this nonconvex problem. A second order Newton type solver is derived to solve the problem. The efficacy of the approach is shown on several examples.

Self-supervised learning of hologram reconstruction using physics consistency

Existing applications of deep learning in computational imaging and microscopy mostly depend on supervised learning, requiring large-scale, diverse and labelled training data. The acquisition and preparation of such training image datasets is often laborious and costly, leading to limited generalization to new sample types. Here we report a self-supervised learning model, termed GedankenNet, that eliminates the need for labelled or experimental training data, and demonstrate its effectiveness and superior generalization on hologram reconstruction tasks. Without prior knowledge about the sample types, the self-supervised learning model was trained using a physics-consistency loss and artificial random images synthetically generated without any experiments or resemblance to real-world samples. After its self-supervised training, GedankenNet successfully generalized to experimental holograms of unseen biological samples, reconstructing the phase and amplitude images of different types of object using experimentally acquired holograms. Without access to experimental data, knowledge of real samples or their spatial features, GedankenNet achieved complex-valued image reconstructions consistent with the wave equation in free space. The GedankenNet framework also shows resilience to random, unknown perturbations in the physical forward model, including changes in the hologram distances, pixel size and illumination wavelength. This self-supervised learning of image reconstruction creates new opportunities for solving inverse problems in holography, microscopy and computational imaging.

One-dimensional convolutional neural network for Jacobian in Diffuse Optical Tomography.

Non-linear least square minimization algorithms are often employed to solve Diffuse Optical Tomography (DOT) inverse problem. However, it is time-consuming to calculate the Jacobian matrix. This work has proposed a data-driven neural network method to improve computational efficiency. The singular value decomposition is employed to compute the updated Jacobian and a mapping from boundary measurements to the singular values based on a convolutional neural network (CNN) is learned to obtain the singular values. The method is validated with 3D numerical simulation data. We have demonstrated that the approach can save computation time compared to Adjoint method, and reconstructed absorption coefficient close to Adjoint method.Clinical Relevance- These results are not focused on clinical relevance currently, but in the future may be helpful to accelerant DOT reconstruction in clinic.

A Positive Data Extraction Method for Electrical Impedance Tomography (EIT) Based on the Novel MSA-Net

The nonlinear and ill-posed Electrical Impedance Tomography (EIT) inverse problem leads to reconstructed images with distortion and low resolution. A positive data extraction method based on the deep learning is proposed in this study. This method realizes the accurate classification of the positive data and the general data in voltage measurements through a deep learning network called Multi Scale Attention Network (MSA-Net). The positive data is used for image reconstruction to improve the imaging quality. Multi scale feature extraction, residual learning and attention mechanism are introduced in the MSA-Net to improve classification accuracy. The Linear Back Projection (LBP) algorithm and the Tikhonov Regularization (TR) algorithm are used to verify the effectiveness of the proposed method. Both simulation and experiment are conducted around imaging of bubble flow. The simulation results indicate that there are fewer artifacts in the reconstructed images using the positive data of MSA-Net, and the shape and position of the bubbles can be better described. Moreover, the imaging quality of experimental samples can also be effectively improved. The existing test results preliminarily confirm that the distribution of the positive data and the general data in voltage measurements is related to the position of bubbles in the measurement domain.

Numerical and experimental investigation of many-objective optimization for efficient temperature and velocity fields reconstruction via acoustic tomography

Accurate and high-quality measurement of temperature and velocity fields is important in combustion and flow diagnosis, especially for the control of boiler operations and optimization of combustion processes. Nonlinear acoustic tomography (NAT) is an appealing approach to simultaneously monitor temperature and velocity fields by observing the changes in sound speed they induce. However, the inherent ill-posedness of the tomographic inverse problem and the lack of a priori information may lead to poor robustness, low precision, and the presence of artifacts. On the other hand, the inclusion of a priori information, such as smoothness and box constraints, may require the use of hyperparameters that hinder quasi real-time reconstruction and may degrade performance in actual applications. In this study, we aim to ensure quasi real-time reconstruction and alleviate the ill-posedness of the NAT problem by eliminating the regularization parameters and transforming the inverse problem into a many-objective optimization problem with four objectives. The knee point-driven evolutionary algorithm with improved environmental selection strategy (KEA-IES) is proposed to simultaneously reconstruct the arbitrary inhomogeneous temperature and aerodynamic fields in a furnace. Apart from the investigation of the performance of KEA-IES and the influence of various factors, including measurement noise, function evaluation number, population size, and functional norms, on the quality of the reconstruction, an experimental AT system with independent 16 T/R channels is developed to evaluate the proposed method. The results show that combustion flame temperature and velocity fields can be monitored simultaneously by using the proposed KEA-IES with more accuracy, less consumed time, and better noise immunity compared with the state-of-the-art algorithms. The proposed method can provide valuable guidance in the development of a non-intrusive real-time pyrometry and velocimetry system, particularly for applications with large velocity fields and temperature gradients. Furthermore, its capability in complex combustion and flow diagnosis will offer benefits in terms of energy conservation, emission reduction, boiler operations supervision, and combustion process optimization.

Ambiguity in Solving Imaging Inverse Problems with Deep-Learning-Based Operators.

In recent years, large convolutional neural networks have been widely used as tools for image deblurring, because of their ability in restoring images very precisely. It is well known that image deblurring is mathematically modeled as an ill-posed inverse problem and its solution is difficult to approximate when noise affects the data. Really, one limitation of neural networks for deblurring is their sensitivity to noise and other perturbations, which can lead to instability and produce poor reconstructions. In addition, networks do not necessarily take into account the numerical formulation of the underlying imaging problem when trained end-to-end. In this paper, we propose some strategies to improve stability without losing too much accuracy to deblur images with deep-learning-based methods. First, we suggest a very small neural architecture, which reduces the execution time for training, satisfying a green AI need, and does not extremely amplify noise in the computed image. Second, we introduce a unified framework where a pre-processing step balances the lack of stability of the following neural-network-based step. Two different pre-processors are presented. The former implements a strong parameter-free denoiser, and the latter is a variational-model-based regularized formulation of the latent imaging problem. This framework is also formally characterized by mathematical analysis. Numerical experiments are performed to verify the accuracy and stability of the proposed approaches for image deblurring when unknown or not-quantified noise is present; the results confirm that they improve the network stability with respect to noise. In particular, the model-based framework represents the most reliable trade-off between visual precision and robustness.

Unsupervised Deep Learning for Phase Retrieval via Teacher-Student Distillation

Phase retrieval (PR) is a challenging nonlinear inverse problem in scientific imaging that involves reconstructing the phase of a signal from its intensity measurements. Recently, there has been an increasing interest in deep learning-based PR. Motivated by the challenge of collecting ground-truth (GT) images in many domains, this paper proposes a fully-unsupervised learning approach for PR, which trains an end-to-end deep model via a GT-free teacher-student online distillation framework. Specifically, a teacher model is trained using a self-expressive loss with noise resistance, while a student model is trained with a consistency loss on augmented data to exploit the teacher's dark knowledge. Additionally, we develop an enhanced unfolding network for both the teacher and student models. Extensive experiments show that our proposed approach outperforms existing unsupervised PR methods with higher computational efficiency and performs competitively against supervised methods.

Adaptive constraints by morphological operations for single-shot digital holography

Digital holography provides access to quantitative measurement of the entire complex field, which is indispensable for the investigation of wave-matter interactions. The emerging iterative phase retrieval approach enables to solve the inverse imaging problem only from the given intensity measurements and physical constraints. However, enforcing imprecise constraints limits the reconstruction accuracy and convergence speed. Here, we propose an advanced iterative phase retrieval framework for single-shot in-line digital holography that incorporates adaptive constraints, which achieves optimized convergence behavior, high-fidelity and twin-image-free reconstruction. In conjunction with morphological operations which can extract the object structure while eliminating the irrelevant part such as artifacts and noise, adaptive constraints allow the support region to be accurately estimated and automatically updated at each iteration. Numerical reconstruction of complex-valued objects and the capability of noise immunity are investigated. The improved reconstruction performance of this approach is experimentally validated. Such flexible and versatile framework has promising applications in biomedicine, X-ray coherent diffractive imaging and wavefront sensing.

A Novel Method for the Numerical Solution of a Hybrid Inverse Problem of Electrical Conductivity Imaging

A novel method for the numerical solution of a hybrid (coupled physics) inverse problem is proposed. Based on a regularized weighted mean curvature flow equation, this method can be considered as an alternative to the variational approach to solving weighted least gradient Dirichlet problems arising in electrical conductivity imaging, in particular, in Current Density Impedance Imaging (CDII). Utilizing the Sternberg–Ziemer theory, convergence of regularized solutions to a unique function of weighted least gradient is established. The numerical study is conducted to demonstrate the practicability and computational effectiveness of the proposed method.

Parametric level-set inverse problems with stochastic background estimation

We study parametric shape reconstruction inverse problems in which the object of interest is embedded in a heterogeneous background medium that is known only approximately. We model the background medium as a Gaussian random field and pose shape reconstruction as a stochastic programming problem in which we seek to minimize the expected value, with respect to the background field, of a stochastic objective function. We develop a computationally efficient algorithm based on the sample average approximation that reduces the effect of uncertainty in the background medium on shape recovery. We demonstrate that by using accelerated stochastic gradient descent, we can apply our method to large-scale problems. The capabilities of our method are demonstrated on a simple two-dimensional model problem and in a more demanding application to a three-dimensional inverse conductivity problem in geophysical imaging.

PatchNR: learning from very few images by patch normalizing flow regularization

Learning neural networks using only few available information is an important ongoing research topic with tremendous potential for applications. In this paper, we introduce a powerful regularizer for the variational modeling of inverse problems in imaging. Our regularizer, called patch normalizing flow regularizer (patchNR), involves a normalizing flow learned on small patches of very few images. In particular, the training is independent of the considered inverse problem such that the same regularizer can be applied for different forward operators acting on the same class of images. By investigating the distribution of patches versus those of the whole image class, we prove that our model is indeed a maximum a posteriori approach. Numerical examples for low-dose and limited-angle computed tomography (CT) as well as superresolution of material images demonstrate that our method provides very high quality results. The training set consists of just six images for CT and one image for superresolution. Finally, we combine our patchNR with ideas from internal learning for performing superresolution of natural images directly from the low-resolution observation without knowledge of any high-resolution image.

Real-time phase imaging with physics-enhanced network and equivariance.

Learning-based phase imaging balances high fidelity and speed. However, supervised training requires unmistakable and large-scale datasets, which are often hard or impossible to obtain. Here, we propose an architecture for real-time phase imaging based on physics-enhanced network and equivariance (PEPI). The measurement consistency and equivariant consistency of physical diffraction images are used to optimize the network parameters and invert the process from a single diffraction pattern. In addition, we propose a regularization method based total variation kernel (TV-K) function constraint to output more texture details and high-frequency information. The results show that PEPI can produce the object phase quickly and accurately, and the proposed learning strategy performs closely to the fully supervised method in the evaluation function. Moreover, the PEPI solution can handle high-frequency details better than the fully supervised method. The reconstruction results validate the robustness and generalization ability of the proposed method. Specially, our results show that PEPI leads to considerable performance improvement on the imaging inverse problem, thereby paving the way for high-precision unsupervised phase imaging.

Lateral Strain Imaging using Self-supervised and Physically Inspired Constraints in Unsupervised Regularized Elastography.

Convolutional Neural Networks (CNN) have shown promising results for displacement estimation in UltraSound Elastography (USE). Many modifications have been proposed to improve the displacement estimation of CNNs for USE in the axial direction. However, the lateral strain, which is essential in several downstream tasks such as the inverse problem of elasticity imaging, remains a challenge. The lateral strain estimation is complicated since the motion and the sampling frequency in this direction are substantially lower than the axial one, and a lack of carrier signal in this direction. In computer vision applications, the axial and the lateral motions are independent. In contrast, the tissue motion pattern in USE is governed by laws of physics which link the axial and lateral displacements. In this paper, inspired by Hooke's law, we first propose Physically Inspired ConsTraint for Unsupervised Regularized Elastography (PICTURE), where we impose a constraint on the Effective Poisson's ratio (EPR) to improve the lateral strain estimation. In the next step, we propose self-supervised PICTURE (sPICTURE) to further enhance the strain image estimation. Extensive experiments on simulation, experimental phantom and in vivo data demonstrate that the proposed methods estimate accurate axial and lateral strain maps.