Image Reconstruction Approach Research Articles

A new ECT image reconstruction algorithm based on Vision transformer (ViT)

Aiming at the problem of low accuracy of ECT image reconstruction, this paper proposes an ECT image reconstruction algorithm based on Vision Transformer (ViT). ViT is a method that applies the Transformer to the field of image classification, which is characterized by strong long-distance dependency learning ability and strong multi-modal fusion ability compared to CNN. This paper fully utilizes the characteristics of ViT to transform the image reconstruction process of ECT into the classification process of ViT. Here, a method is proposed to use the object field distribution as the image classification label. This method converts a two-dimensional image of the object field distribution into a one-dimensional vector, which is the label vector of the image classification. These vectors and their corresponding reconstructed images obtained by the Landweber algorithm form the sample set of ViT. Extracting a large number of flow pattern samples with various shapes and distributions through COMSOL is used as a training set. After training ViT, this network model is used to infer the labels of the predicted flow patterns. After post-processing this label, the corresponding reconstructed image can be obtained. Finally, simulation experiments are conducted, and the experiments results show that the image errors and correlation coefficients of the reconstructed images obtained through the algorithm in this paper are better than those of Tikhonov algorithm, Landweber algorithm and Long Short-Term Memory Network(LSTM). And this algorithm has better resistance to noise interference than Tikhonov algorithm, Landweber algorithm, and LSTM. This also provides a new approach and means for ECT image reconstruction.

Joint reconstruction and segmentation in undersampled 3D knee MRI combining shape knowledge and deep learning

Objective. Task-adapted image reconstruction methods using end-to-end trainable neural networks (NNs) have been proposed to optimize reconstruction for subsequent processing tasks, such as segmentation. However, their training typically requires considerable hardware resources and thus, only relatively simple building blocks, e.g. U-Nets, are typically used, which, albeit powerful, do not integrate model-specific knowledge. Approach. In this work, we extend an end-to-end trainable task-adapted image reconstruction method for a clinically realistic reconstruction and segmentation problem of bone and cartilage in 3D knee MRI by incorporating statistical shape models (SSMs). The SSMs model the prior information and help to regularize the segmentation maps as a final post-processing step. We compare the proposed method to a simultaneous multitask learning approach for image reconstruction and segmentation (MTL) and to a complex SSMs-informed segmentation pipeline (SIS). Main results. Our experiments show that the combination of joint end-to-end training and SSMs to further regularize the segmentation maps obtained by MTL highly improves the results, especially in terms of mean and maximal surface errors. In particular, we achieve the segmentation quality of SIS and, at the same time, a substantial model reduction that yields a five-fold decimation in model parameters and a computational speedup of an order of magnitude. Significance. Remarkably, even for undersampling factors of up to R = 8, the obtained segmentation maps are of comparable quality to those obtained by SIS from ground-truth images.

Effect of MR head coil geometry on deep-learning-based MR image reconstruction.

To investigate whether parallel imaging-imposed geometric coil constraints can be relaxed when using a deep learning (DL)-based image reconstruction method as opposed to a traditional non-DL method. Traditional and DL-based MR image reconstruction approaches operate in fundamentally different ways: Traditional methods solve a system of equations derived from the image data whereas DL methods use data/target pairs to learn a generalizable reconstruction model. Two sets of head coil profiles were evaluated: (1) 8-channel and (2) 32-channel geometries. A DL model was compared to conjugate gradient SENSE (CG-SENSE) and L1-wavelet compressed sensing (CS) through quantitative metrics and visual assessment as coil overlap was increased. Results were generally consistent between experiments. As coil overlap increased, there was a significant (p < 0.001) decrease in performance in most cases for all methods. The decrease was most pronounced for CG-SENSE, and the DL models significantly outperformed (p < 0.001) their non-DL counterparts in all scenarios. CS showed improved robustness to coil overlap and signal-to-noise ratio (SNR) versus CG-SENSE, but had quantitatively and visually poorer reconstructions characterized by blurriness as compared to DL. DL showed virtually no change in performance across SNR and very small changes across coil overlap. The DL image reconstruction method produced images that were robust to coil overlap and of higher quality than CG-SENSE and CS. This suggests that geometric coil design constraints can be relaxed when using DL reconstruction methods.

Task-Free Continual Generation and Representation Learning via Dynamic Expansionable Memory Cluster

Human brains can continually acquire and learn new skills and knowledge over time from a dynamically changing environment without forgetting previously learnt information. Such a capacity can selectively transfer some important and recently seen information to the persistent knowledge regions of the brain. Inspired by this intuition, we propose a new memory-based approach for image reconstruction and generation in continual learning, consisting of a temporary and evolving memory, with two different storage strategies, corresponding to the temporary and permanent memorisation. The temporary memory aims to preserve up-to-date information while the evolving memory can dynamically increase its capacity in order to preserve permanent knowledge information. This is achieved by the proposed memory expansion mechanism that selectively transfers those data samples deemed as important from the temporary memory to new clusters defined within the evolved memory according to an information novelty criterion. Such a mechanism promotes the knowledge diversity among clusters in the evolved memory, resulting in capturing more diverse information by using a compact memory capacity. Furthermore, we propose a two-step optimization strategy for training a Variational Autoencoder (VAE) to implement generation and representation learning tasks, which updates the generator and inference models separately using two optimisation paths. This approach leads to a better trade-off between generation and reconstruction performance. We show empirically and theoretically that the proposed approach can learn meaningful latent representations while generating diverse images from different domains. The source code and supplementary material (SM) are available at https://github.com/dtuzi123/DEMC.

Optimal image 2D/3D by Krawtchouk moments and ABC algorithm

In our research, we provide an enhanced and effective Krawtchouk moments parameter p optimization approach using the Artificial Bee Colony (ABC) algorithm optimization technique for the purpose of completing tasks including the reconstruction and classification of 3D and 2D images with high quality. The proposed method, which uses the ABC algorithm to generate a suitable parameter based on Krawtchouck moments, aims to accurately characterize the image as a vector known as the descriptor vector of moments, which is then used to reconstruct the image for each order. The simulation and analysis of the findings demonstrate the high level of quality and effectiveness of the suggested KM-ABC approach for image reconstruction, as well as its excellent accuracy in images with and without noise.

Reusability report: Unpaired deep-learning approaches for holographic image reconstruction

Deep-learning methods using unpaired datasets hold great potential for image reconstruction, especially in biomedical imaging where obtaining paired datasets is often difficult due to practical concerns. A recent study by Lee et al. (Nature Machine Intelligence 2023) has introduced a parameterized physical model (referred to as FMGAN) using the unpaired approach for adaptive holographic imaging, which replaces the forward generator network with a physical model parameterized on the propagation distance of the probing light. FMGAN has demonstrated its capability to reconstruct the complex phase and amplitude of objects, as well as the propagation distance, even in scenarios where the object-to-sensor distance exceeds the range of the training data. We performed additional experiments to comprehensively assess FMGAN’s capabilities and limitations. As in the original paper, we compared FMGAN to two state-of-the-art unpaired methods, CycleGAN and PhaseGAN, and evaluated their robustness and adaptability under diverse conditions. Our findings highlight FMGAN’s reproducibility and generalizability when dealing with both in-distribution and out-of-distribution data, corroborating the results reported by the original authors. We also extended FMGAN with explicit forward models describing the response of specific optical systems, which improved performance when dealing with non-perfect systems. However, we observed that FMGAN encounters difficulties when explicit forward models are unavailable. In such scenarios, PhaseGAN outperformed FMGAN.

Quantum annealing-based computed tomography using variational approach for a real-number image reconstruction

Objective. Despite recent advancements in quantum computing, the limited number of available qubits has hindered progress in CT reconstruction. This study investigates the feasibility of utilizing quantum annealing-based computed tomography (QACT) with current quantum bit levels. Approach. The QACT algorithm aims to precisely solve quadratic unconstrained binary optimization problems. Furthermore, a novel approach is proposed to reconstruct images by approximating real numbers using the variational method. This approach allows for accurate CT image reconstruction using a small number of qubits. The study examines the impact of projection data quantity and noise on various image sizes ranging from 4 × 4 to 24 × 24 pixels. The reconstructed results are compared against conventional reconstruction algorithms, namely maximum likelihood expectation maximization (MLEM) and filtered back projection (FBP). Main result. By employing the variational approach and utilizing two qubits for each pixel of the image, accurate reconstruction was achieved with an adequate number of projections. Under conditions of abundant projections and lower noise levels, the image quality in QACT algorithm outperformed that of MLEM and FBP algorithms. However, in situations with limited projection data and in the presence of noise, the image quality in QACT was inferior to that in MLEM. Significance. This study developed the QACT reconstruction algorithm using the variational approach for real-number reconstruction. Remarkably, only 2 qubits were required for each pixel representation, demonstrating their sufficiency for accurate reconstruction.

Deep image prior inpainting of ancient frescoes in the Mediterranean Alpine arc

The unprecedented success of image reconstruction approaches based on deep neural networks has revolutionised both the processing and the analysis paradigms in several applied disciplines. In the field of digital humanities, the task of digital reconstruction of ancient frescoes is particularly challenging due to the scarce amount of available training data caused by ageing, wear, tear and retouching over time. To overcome these difficulties, we consider the Deep Image Prior (DIP) inpainting approach which computes appropriate reconstructions by relying on the progressive updating of an untrained convolutional neural network so as to match the reliable piece of information in the image at hand while promoting regularisation elsewhere. In comparison with state-of-the-art approaches (based on variational/PDEs and patch-based methods), DIP-based inpainting reduces artefacts and better adapts to contextual/non-local information, thus providing a valuable and effective tool for art historians. As a case study, we apply such approach to reconstruct missing image contents in a dataset of highly damaged digital images of medieval paintings located into several chapels in the Mediterranean Alpine Arc and provide a detailed description on how visible and invisible (e.g., infrared) information can be integrated for identifying and reconstructing damaged image regions.

Perception and memory-based representations of facial emotions: Associations with personality functioning, affective states and recognition abilities

Personality traits and affective states are associated with biases in facial emotion perception. However, the precise personality impairments and affective states that underlie these biases remain largely unknown. To investigate how relevant factors influence facial emotion perception and recollection, Experiment 1 employed an image reconstruction approach in which community-dwelling adults (N = 89) rated the similarity of pairs of facial expressions, including those recalled from memory. Subsequently, perception- and memory-based expression representations derived from such ratings were assessed across participants and related to measures of personality impairment, state affect, and visual recognition abilities. Impairment in self-direction and level of positive affect accounted for the largest components of individual variability in perception and memory representations, respectively. Additionally, individual differences in these representations were impacted by face recognition ability. In Experiment 2, adult participants (N = 81) rated facial image reconstructions derived in Experiment 1, revealing that individual variability was associated with specific visual face properties, such as expressiveness, representation accuracy, and positivity/negativity. These findings highlight and clarify the influence of personality, affective state, and recognition abilities on individual differences in the perception and recollection of facial expressions.

Exploiting sparsity of hyperspectral image: A novel approach for compressive hyperspectral image reconstruction using deep learning

Compressive hyperspectral imaging is an emerging technology that captures compressed two-dimensional (2D) measurements and subsequently reconstructs three-dimensional (3D) hyperspectral images, capitalizing on the sparsity prior. Numerous end-to-end networks have been proposed, yielding notable improvements in reconstruction speed and quality compared to traditional algorithms. However, these networks fall short in exploiting the sparsity of hyperspectral images in their direct mapping from 2D measurements to 3D hyperspectral images. To address this limitation, this paper introduces a novel approach for compressive hyperspectral image reconstruction using deep learning. The deep learning network comprises two components: the sparse coefficient recovery net and the dictionary training net. The dictionary training net incorporates a pre-trained dictionary, whose parameters can also be updated during the backpropagation process. Concurrently, the sparse coefficient recovery net generates sparse coefficients under this dictionary. Our approach draws inspiration from conventional dictionary learning algorithms, enabling effective utilization of signal sparsity. In comparison to several state-of-the-art methods, our method achieves superior reconstruction quality and demonstrates enhanced robustness against noise. The proposed approach is envisioned to offer fresh insights into hyperspectral images reconstruction using deep learning models.

Image reconstruction techniques using deep learning quality segmentation

Translational CT (TCT), in developing nations, a low-end computed tomography (CT) technology are relatively common. The limited-angle TCT scanning mode is often used with large-angle scanning to scan items within a narrow angular range, reduce X-ray radiation, scan long objects, and prevent detector discrepancies.. However, this scanning mode greatly reduces the picture quality and diagnostic accuracy due to the added noise and limited-angle distortions. A U-net convolutional neural network-based approach for limited-angle TCT image reconstruction has been created to reconstruct a high-quality image for the limited-angle TCT scanning mode (CNN). The limited-angle TCT projection data are first examined using the SART method, and the resulting picture is then fed into a trained CNN that can reduce artifacts and maintain structures to provide a better reconstructed image. Simulated studies are used to demonstrate the effectiveness of the algorithm designed for the limitedangle TCT scanning mode. In contrast to certain modern techniques, the developed algorithm considerably lowers noise and limited-angle artifacts while maintaining image structures.

NeRP: Implicit Neural Representation Learning With Prior Embedding for Sparsely Sampled Image Reconstruction.

Image reconstruction is an inverse problem that solves for a computational image based on sampled sensor measurement. Sparsely sampled image reconstruction poses additional challenges due to limited measurements. In this work, we propose a methodology of implicit Neural Representation learning with Prior embedding (NeRP) to reconstruct a computational image from sparsely sampled measurements. The method differs fundamentally from previous deep learning-based image reconstruction approaches in that NeRP exploits the internal information in an image prior and the physics of the sparsely sampled measurements to produce a representation of the unknown subject. No large-scale data is required to train the NeRP except for a prior image and sparsely sampled measurements. In addition, we demonstrate that NeRP is a general methodology that generalizes to different imaging modalities such as computed tomography (CT) and magnetic resonance imaging (MRI). We also show that NeRP can robustly capture the subtle yet significant image changes required for assessing tumor progression.

Joint B0 and Image Reconstruction in Low-Field MRI by Physics-Informed Deep-Learning.

We present a model-based image reconstruction approach based on unrolled neural networks which corrects for image distortion and noise in low-field ( B0 ∼ 50mT) MRI. Utilising knowledge about the underlying physics, a novel network architecture (SH-Net) is introduced which involves the estimation of spherical harmonic coefficients to guarantee a spatially smooth field map estimate. The SH-Net is integrated in an end-to-end trainable model which jointly estimates the B0-field map as well as the image. Experiments were conducted on retrospectively simulated low-field data of human knees. We compare our model to different model-based approaches at distinct noise levels and various B0-field distributions. Our results show that our physics-informed neural network approach outperforms the purely model-based methods by improving the PSNR up to 11.7% and the RMSE up to 86.3%. Our end-to-end trained model-based approach outperforms existing methods in reconstructing image and B0-field maps in the low-field regime. low-field MRI is becoming increasingly more popular as it enables access to MR in challenging situations such as intensive care units or resource poor areas. Our method allows for fast and accurate image reconstruction in such low-field imaging with B0-inhomogeneity compensation under a wide range of various environmental conditions.

A Cloud Coverage Image Reconstruction Approach for Remote Sensing of Temperature and Vegetation in Amazon Rainforest

Remote sensing involves actions to obtain information about an area located on Earth. In the Amazon region, the presence of clouds is a common occurrence, and the visualization of important terrestrial information in the image, like vegetation and temperature, can be difficult. In order to estimate land surface temperature (LST) and the normalized difference vegetation index (NDVI) from satellite images with cloud coverage, the inpainting approach will be applied to remove clouds and restore the image of the removed region. This paper proposes the use of the neural network LaMa (large mask inpainting) and the scalable model named Big LaMa for the automatic reconstruction process in satellite images. Experiments are conducted on Landsat-8 satellite images of the Amazon rainforest in the state of Acre, Brazil. To evaluate the architecture’s accuracy, the RMSE (root mean squared error), SSIM (structural similarity index) and PSNR (peak signal-to-noise ratio) metrics were used. The LST and NDVI of the reconstructed image were calculated and compared qualitatively and quantitatively, using scatter plots and the chosen metrics, respectively. The experimental results show that the Big LaMa architecture performs more effectively and robustly in restoring images in terms of visual quality. And the LaMa network shows minimal superiority for the measured metrics when addressing medium marked areas. When comparing the results achieved in NDVI and LST of the reconstructed images with real cloud coverage, great visual results were obtained with Big LaMa.

Electrical impedance tomography for pulmonary function monitoring without dorsal electrodes

Electrical impedance tomography (EIT) is a non-invasive technology that can visualize conductivity changes inside human body in real time using multiple surface electrodes. For convenience of wearing and application, if no electrodes are attached to the back, a commonly used image reconstruction approach produces poor images of dorsal region. In this study, we developed a special current injection and voltage measurement pattern to well reconstruct the conductivity distribution inside the body even in the absence of dorsal electrodes. The proposed method has proven through numerical and phantom experiments.

Transformer-Based Integrated Framework for Joint Reconstruction and Segmentation in Accelerated Knee MRI

Magnetic Resonance Imaging (MRI) reconstruction and segmentation are crucial for medical diagnostics and treatment planning. Despite advances, achieving high performance in both tasks remains challenging, especially in the context of accelerated MRI acquisition. Motivated by this challenge, the objective of this study is to develop an integrated approach for MRI image reconstruction and segmentation specifically tailored for accelerated acquisition scenarios. The proposed method unifies these tasks by incorporating segmentation feedback into an iterative reconstruction algorithm and using a transformer-based encoder–decoder architecture. This architecture consists of a shared encoder and task-specific decoders, and employs a feature distillation process between the decoders. The proposed model is evaluated on the Stanford Knee MRI with Multi-Task Evaluation (SKM-TEA) dataset against established methods such as SegNetMRI and IDSLR-Seg. The results show improvements in the PSNR, SSIM, Dice, and Hausdorff distance metrics. An ablation study confirms the contribution of feature distillation and segmentation feedback to the performance gains. The advancements demonstrated in this study have the potential to impact clinical practice by facilitating more accurate diagnosis and better-informed treatment plans.

Efficient physics-based learned reconstruction methods for real-time 3D near-field MIMO radar imaging

Near-field multiple-input multiple-output (MIMO) radar imaging systems have recently gained significant attention. These systems generally reconstruct the three-dimensional (3D) complex-valued reflectivity distribution of the scene using sparse measurements. Consequently, imaging quality highly relies on the image reconstruction approach. Existing analytical reconstruction approaches suffer from either high computational cost or low image quality. In this paper, we develop novel non-iterative deep learning-based reconstruction methods for real-time near-field MIMO imaging. The goal is to achieve high image quality with low computational cost at compressive settings. The developed approaches have two stages. In the first approach, physics-based initial stage performs adjoint operation to back-project the measurements to the image-space, and deep neural network (DNN)-based second stage converts the 3D backprojected measurements to a magnitude-only reflectivity image. Since scene reflectivities often have random phase, DNN processes directly the magnitude of the adjoint result. As DNN, 3D U-Net is used to jointly exploit range and cross-range correlations. To comparatively evaluate the significance of exploiting physics in a learning-based approach, two additional approaches that replace the physics-based first stage with fully connected layers are also developed as purely learning-based methods. The performance is also analyzed by changing the DNN architecture for the second stage to include complex-valued processing (instead of magnitude-only processing), 2D convolution kernels (instead of 3D), and ResNet architecture (instead of U-Net). Moreover, we develop a synthesizer to generate large-scale dataset for training the neural networks with 3D extended targets. We illustrate the performance through experimental data and extensive simulations. The results show the effectiveness of the developed physics-based learned reconstruction approach compared to commonly used approaches in terms of both runtime and image quality at highly compressive settings. Our source codes and dataset are made available at https://github.com/METU-SPACE-Lab/Efficient-Learned-3D-Near-Field-MIMO-Imaging upon publication to advance research in this field.

U-shaped fusion convolutional transformer based workflow for fast optical coherence tomography angiography generation in lips.

Oral disorders, including oral cancer, pose substantial diagnostic challenges due to late-stage diagnosis, invasive biopsy procedures, and the limitations of existing non-invasive imaging techniques. Optical coherence tomography angiography (OCTA) shows potential in delivering non-invasive, real-time, high-resolution vasculature images. However, the quality of OCTA images are often compromised due to motion artifacts and noise, necessitating more robust and reliable image reconstruction approaches. To address these issues, we propose a novel model, a U-shaped fusion convolutional transformer (UFCT), for the reconstruction of high-quality, low-noise OCTA images from two-repeated OCT scans. UFCT integrates the strengths of convolutional neural networks (CNNs) and transformers, proficiently capturing both local and global image features. According to the qualitative and quantitative analysis in normal and pathological conditions, the performance of the proposed pipeline outperforms that of the traditional OCTA generation methods when only two repeated B-scans are performed. We further provide a comparative study with various CNN and transformer models and conduct ablation studies to validate the effectiveness of our proposed strategies. Based on the results, the UFCT model holds the potential to significantly enhance clinical workflow in oral medicine by facilitating early detection, reducing the need for invasive procedures, and improving overall patient outcomes.

Optimization of 3D dynamic speech MRI: Poisson-disc undersampling and locally higher-rank reconstruction through partial separability model with regional optimized temporal basis.

To improve the spatiotemporal qualities of images and dynamics of speech MRI through an improved data sampling and image reconstruction approach. For data acquisition, we used a Poisson-disc random under sampling scheme that reduced the undersampling coherence. For image reconstruction, we proposed a novel locally higher-rank partial separability model. This reconstruction model represented the oral and static regions using separate low-rank subspaces, therefore, preserving their distinct temporal signal characteristics. Regional optimized temporal basis was determined from the regional-optimized virtual coil approach. Overall, we achieved a better spatiotemporal image reconstruction quality with the potential of reducing total acquisition time by 50%. The proposed method was demonstrated through several 2-mm isotropic, 64 mm total thickness, dynamic acquisitions with 40 frames per second and compared to the previous approach using a global subspace model along with other k-space sampling patterns. Individual timeframe images and temporal profiles of speech samples were shown to illustrate the ability of the Poisson-disc under sampling pattern in reducing total acquisition time. Temporal information of sagittal and coronal directions was also shown to illustrate the effectiveness of the locally higher-rank operator and regional optimized temporal basis. To compare the reconstruction qualities of different regions, voxel-wise temporal SNR analysis were performed. Poisson-disc sampling combined with a locally higher-rank model and a regional-optimized temporal basis can drastically improve the spatiotemporal image quality and provide a 50% reduction in overall acquisition time.

Mitigating the Limited View Problem in Photoacoustic Tomography for a Planar Detection Geometry by Regularized Iterative Reconstruction.

The use of a planar detection geometry in photoacoustic tomography results in the so- called limited-view problem due to the finite extent of the acoustic detection aperture. When images are reconstructed using one-step reconstruction algorithms, image quality is compromised by the presence of streaking artefacts, reduced contrast, image distortion and reduced signal-to-noise ratio. To mitigate this, model-based iterative reconstruction approaches based on least squares minimisation with and without total variation regularization were evaluated using in-silico, experimental phantom, ex vivo and in vivo data. Compared to one-step reconstruction methods, it has been shown that iterative methods provide better image quality in terms of enhanced signal-to-artefact ratio, signal-to-noise ratio, amplitude accuracy and spatial fidelity. For the total variation approaches, the impact of the regularization parameter on image feature scale and amplitude distribution was evaluated. In addition, the extent to which the use of Bregman iterations can compensate for the systematic amplitude bias introduced by total variation was studied. This investigation is expected to inform the practical application of model-based iterative image reconstruction approaches for improving photoacoustic image quality when using finite aperture planar detection geometries.