On Hallucinations in Tomographic Image Reconstruction.

Sayantan Bhadra,Varun A. Kelkar,Frank J. Brooks,Mark A. Anastasio

doi:10.1109/tmi.2021.3077857

Abstract

Tomographic image reconstruction is generally an ill-posed linear inverse problem. Such ill-posed inverse problems are typically regularized using prior knowledge of the sought-after object property. Recently, deep neural networks have been actively investigated for regularizing image reconstruction problems by learning a prior for the object properties from training images. However, an analysis of the prior information learned by these deep networks and their ability to generalize to data that may lie outside the training distribution is still being explored. An inaccurate prior might lead to false structures being hallucinated in the reconstructed image and that is a cause for serious concern in medical imaging. In this work, we propose to illustrate the effect of the prior imposed by a reconstruction method by decomposing the image estimate into generalized measurement and null components. The concept of a hallucination map is introduced for the general purpose of understanding the effect of the prior in regularized reconstruction methods. Numerical studies are conducted corresponding to a stylized tomographic imaging modality. The behavior of different reconstruction methods under the proposed formalism is discussed with the help of the numerical studies.

Highlights

I N tomographic imaging, a reconstruction method is employed to estimate the sought-after object from a collection of measurements obtained from an imaging system [1]
This study proposes a way to mathematically formalize the concept of hallucinations for general linear imaging systems that is consistent with both the mathematical notion of a hallucination in image super-resolution and the intuitive notion of hallucinations as “artifacts or incorrect features that occur due to the prior that cannot be produced from the measurements”
An illustration of hallucination maps is provided for different reconstruction methods, in order to demonstrate their utility in highlighting false structures that may be introduced due to the imposed prior

Summary

INTRODUCTION

I N tomographic imaging, a reconstruction method is employed to estimate the sought-after object from a collection of measurements obtained from an imaging system [1]. The potential lack of generalization of deep learning-based reconstruction methods as well as their innate unstable nature may cause false structures to appear in the reconstructed image that are absent in the object being imaged. These false structures may arise due to the reconstruction method incorrectly estimating parts of the object that either did not contribute to the observed measurement data or cannot be recovered in a stable manner, a phenomenon that can be termed as hallucination.

Imaging models

Generalized measurement and null components

Regularization in tomographic image reconstruction

DEFINITION OF HALLUCINATION MAPS

Hallucination map in the generalized measurement space

Hallucination map in the generalized null space

Specific hallucination maps

NUMERICAL STUDIES

Stylized imaging system

Reconstruction methods

Computation of hallucination maps

RESULTS