Image Reconstruction Task Research Articles

Unified and Scalable Deep Image Compression Framework for Human and Machine

Image compression aims to minimize the amount of data in image representation while maintaining a certain visual quality for humans, which is an essential technique for storage and transmission. Recently, along with the development of computer vision, machines have become another primary receiver for images and require compressed images at a certain quality level, which may be different from that of human vision. In many scenarios, compressed images should serve both human and machine vision tasks, but few compression methods are designed for both goals simultaneously. In this article, we propose a unified and scalable deep image compression (USDIC) framework that jointly optimizes the image quality according to human and machine vision in an end-to-end style. For the encoder, we propose an information splitting mechanism (ISM) to separate images into semantic and visual features, which mainly aims at machine analysis and human viewing tasks. For the decoder, we design a scalable decoding architecture. The encoded semantic feature is first decoded for machine analysis tasks, and the image is decoded and reconstructed further by leveraging the decoded semantic features. Herein, to further remove the redundancy between the semantic and visual features of images, we propose a scalable entropy model (SEM) with a joint optimization strategy to reconstruct the image using the two kinds of decoded features. Extensive experimental results show that the proposed USDIC achieves much better performance on the image analysis task while maintaining competitive performance on the traditional image reconstruction task compared with popular image compression methods.

PET image reconstruction using weighted nuclear norm maximization and deep learning prior

The ill-posed Positron emission tomography (PET) reconstruction problem usually results in limited resolution and significant noise. Recently, deep neural networks have been incorporated into PET iterative reconstruction framework to improve the image quality. In this paper, we propose a new neural network-based iterative reconstruction method by using weighted nuclear norm (WNN) maximization, which aims to recover the image details in the reconstruction process. The novelty of our method is the application of WNN maximization rather than WNN minimization in PET image reconstruction. Meanwhile, a neural network is used to control the noise originated from WNN maximization. Our method is evaluated on simulated and clinical datasets. The simulation results show that the proposed approach outperforms state-of-the-art neural network-based iterative methods by achieving the best contrast/noise tradeoff with a remarkable contrast improvement on the lesion contrast recovery. The study on clinical datasets also demonstrates that our method can recover lesions of different sizes while suppressing noise in various low-dose PET image reconstruction tasks. Our code is available athttps://github.com/Kuangxd/PETReconstruction.

DCE-MRI of the liver with sub-second temporal resolution using GRASP-Pro with navi-stack-of-stars sampling.

Respiratory motion-induced image blurring and artifacts can compromise image quality in dynamic contrast-enhanced MRI (DCE-MRI) of the liver. Despite remarkable advances in respiratory motion detection and compensation in past years, these techniques have not yet seen widespread clinical adoption. The accuracy of image-based motion detection can be especially compromised in the presence of contrast enhancement and/or in situations involving deep and/or irregular breathing patterns. This work proposes a framework that combines GRASP-Pro (Golden-angle RAdial Sparse Parallel MRI with imProved performance) MRI with a new radial sampling scheme called navi-stack-of-stars for free-breathing DCE-MRI of the liver without the need for explicit respiratory motion compensation. A prototype 3D golden-angle radial sequence with a navi-stack-of-stars sampling scheme that intermittently acquires a 2D navigator was implemented. Free-breathing DCE-MRI of the liver was conducted in 24 subjects at 3T including 17 volunteers and 7 patients. GRASP-Pro reconstruction was performed with a temporal resolution of 0.34-0.45 s per volume, whereas standard GRASP reconstruction was performed with a temporal resolution of 15 s per volume. Motion compensation was not performed in all image reconstruction tasks. Liver images in different contrast phases from both GRASP and GRASP-Pro reconstructions were visually scored by two experienced abdominal radiologists for comparison. The nonparametric paired two-tailed Wilcoxon signed-rank test was used to compare image quality scores, and the Cohen's kappa coefficient was calculated to evaluate the inter-reader agreement. GRASP-Pro MRI with sub-second temporal resolution consistently received significantly higher image quality scores (P < 0.05) than standard GRASP MRI throughout all contrast enhancement phases and across all assessment categories. There was a substantial inter-reader agreement for all assessment categories (ranging from 0.67 to 0.89). The proposed technique using GRASP-Pro reconstruction with navi-stack-of-stars sampling holds great promise for free-breathing DCE-MRI of the liver without respiratory motion compensation.

Accurate and fast quaternion fractional-order Franklin moments for color image analysis

Quaternion type moments (QTM) based on hypergeometric functions and harmonic functions have attracted interest in the imaging field. The Franklin system is a complete orthonormal system consisting of piecewise linear continuous functions with Haar wavelet collocation points. However, due to the lack of efficient computational methods and explicit expressions, research on piecewise polynomials with Haar wavelet collocation points remains underexplored. Furthermore, the impact of magnitude and phase of quaternion on the color image features extracted by QTM has not been comprehensively explored. To address these problems, a fast and accurate computation method for Franklin functions is proposed. Quaternion fractional-order Franklin moments (QFFM) are proposed and using the Minkowski distance as a prior parameter for selecting unit pure quaternions. Experiments are conducted to demonstrate the superior performance of QFFM in image reconstruction and color image classification tasks compared with QTM-based and deep learning-based methods. Moreover, with the help of experiments, it is confirmed the effectiveness of using the Minkowski distance to select magnitude and phase of quaternion.

Enhancing global sensitivity and uncertainty quantification in medical image reconstruction with Monte Carlo arbitrary-masked mamba

Deep learning has been extensively applied in medical image reconstruction, where Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) represent the predominant paradigms, each possessing distinct advantages and inherent limitations: CNNs exhibit linear complexity with local sensitivity, whereas ViTs demonstrate quadratic complexity with global sensitivity. The emerging Mamba has shown superiority in learning visual representation, which combines the advantages of linear scalability and global sensitivity. In this study, we introduce MambaMIR, an Arbitrary-Masked Mamba-based model with wavelet decomposition for joint medical image reconstruction and uncertainty estimation. A novel Arbitrary Scan Masking (ASM) mechanism “masks out” redundant information to introduce randomness for further uncertainty estimation. Compared to the commonly used Monte Carlo (MC) dropout, our proposed MC-ASM provides an uncertainty map without the need for hyperparameter tuning and mitigates the performance drop typically observed when applying dropout to low-level tasks. For further texture preservation and better perceptual quality, we employ the wavelet transformation into MambaMIR and explore its variant based on the Generative Adversarial Network, namely MambaMIR-GAN. Comprehensive experiments have been conducted for multiple representative medical image reconstruction tasks, demonstrating that the proposed MambaMIR and MambaMIR-GAN outperform other baseline and state-of-the-art methods in different reconstruction tasks, where MambaMIR achieves the best reconstruction fidelity and MambaMIR-GAN has the best perceptual quality. In addition, our MC-ASM provides uncertainty maps as an additional tool for clinicians, while mitigating the typical performance drop caused by the commonly used dropout.

Hybrid Neural State-Space Modeling for Supervised and Unsupervised Electrocardiographic Imaging.

State-space modeling (SSM) provides a general framework for many image reconstruction tasks. Error in a priori physiological knowledge of the imaging physics, can bring incorrectness to solutions. Modern deep-learning approaches show great promise but lack interpretability and rely on large amounts of labeled data. In this paper, we present a novel hybrid SSM framework for electrocardiographic imaging (ECGI) to leverage the advantage of state-space formulations in data-driven learning. We first leverage the physics-based forward operator to supervise the learning. We then introduce neural modeling of the transition function and the associated Bayesian filtering strategy. We applied the hybrid SSM framework to reconstruct electrical activity on the heart surface from body-surface potentials. In unsupervised settings of both in-silico and in-vivo data without cardiac electrical activity as the ground truth to supervise the learning, we demonstrated improved ECGI performances of the hybrid SSM framework trained from a small number of ECG observations in comparison to the fixed SSM. We further demonstrated that, when in-silico simulation data becomes available, mixed supervised and unsupervised training of the hybrid SSM achieved a further 40.6% and 45.6% improvements, respectively, in comparison to traditional ECGI baselines and supervised data-driven ECGI baselines for localizing the origin of ventricular activations in real data.

Image reconstruction of electrostatic tomography based on the improved residual network.

The core of electrostatic tomography (EST) is to solve the inverse problem, but the EST independent measurement data are much smaller than the value that needs to be reconstructed, resulting in a more serious inverse problem. This paper presents an improved ResNet-34 network (P-ResNet), which consists of an input layer, a residual feature extraction layer, and an output layer. The number of residual blocks is 3, 4, 4, and 3. After the second convolution in the main path of each residual block, a ReLU activation function is added to enhance the nonlinear expression ability of the network, and the generalization ability of the model is improved by introducing the L2 regularization loss function. A total of 15 930 sets of samples were simulated for the simulation test. After 200 rounds of iteration, the reconstruction results show that the network achieves high accuracy in EST image reconstruction tasks. In addition, the model is tested under different degrees of Gaussian white noise to verify its anti-noise ability. Compared with the traditional algorithms, the image correlation coefficients of this proposed model network are higher. In addition, this paper designs a small sensor to obtain the induced charge values through the principle of electrostatic induction. The reconstructed results obtained from the experimental data are consistent with the simulation results, which verifies the effectiveness and generalization ability of the proposed model.

Efficient blind super-resolution imaging via adaptive degradation-aware estimation

Single image super-resolution (SISR) has achieved prominent success based on deep learning. However, most SISR methods based on the specific degradation pattern, e.g., bicubic interpolation with a bulky network structure, are unsuitable for real-world images reconstructed on edge devices. To mitigate the above limitations, unlike the majority of blind super-resolution methods, which consist of a separate complex degradation kernel projection network and a SR reconstruction network, we propose an end-to-end lightweight multi-degradation oriented network that not only can estimate degradation kernel but also reconstructs HR image fast and accurately. The core components of the designed network adopt a lightweight SR model with region non-local feature similarity learning, i.e., Region Non-Local Feature Block (RNLFB), to establish region-wise global feature correlation. Concretely, more and fewer RNLFBs are used to learn more sophisticated feature representation and relatively simple degradation representation. Then, a degradation kernel projector is equipped to fulfill adaptive degradation-aware estimation from the degradation representation. With the help of the encoded degradation kernel, the reconstruction model can learn to restore a high-resolution image from the feature representation. By doing so, the degradation kernel estimation task and the HR image reconstruction task can be accomplished without designing two complex networks separately, which guides more stable and effective network training under low computational resource costs. Extensive experiments on synthetic and real-world datasets demonstrate that our method can fulfill precise degradation kernel prediction and HR image reconstruction compared with other state-of-the-art SR methods with lower model complexity.

No-Reference Hyperspectral Image Quality Assessment via Ranking Feature Learning

In hyperspectral image (HSI) reconstruction tasks, due to the lack of ground truth in real imaging processes, models are usually trained and validated on simulation datasets and then tested on real measurements captured by real HSI imaging systems. However, due to the gap between the simulation imaging process and the real imaging process, the best model validated on the simulation dataset may fail on real measurements. To obtain the best model for the real-world task, it is crucial to design a suitable no-reference HSI quality assessment metric to reflect the reconstruction performance of different models. In this paper, we propose a novel no-reference HSI quality assessment metric via ranking feature learning (R-NHSIQA), which calculates the Wasserstein distance between the distribution of the deep features of the reconstructed HSIs and the benchmark distribution. Additionally, by introducing the spectral self-attention mechanism, we propose a Spectral Transformer (S-Transformer) to extract the spatial-spectral representative deep features of HSIs. Furthermore, to extract quality-sensitive deep features, we use quality ranking as a pre-training task to enhance the representation capability of the S-Transformer. Finally, we introduce the Wasserstein distance to measure the distance between the distribution of the deep features and the benchmark distribution, improving the assessment capacity of our method, even with non-overlapping distributions. The experimental results demonstrate that the proposed metric yields consistent results with multiple full-reference image quality assessment (FR-IQA) metrics, validating the idea that the proposed metric can serve as a substitute for FR-IQA metrics in real-world tasks.

Diminishing Batch Normalization.

In this article, we propose a generalization of the batch normalization (BN) algorithm, diminishing BN (DBN), where we update the BN parameters in a diminishing moving average way. BN is very effective in accelerating the convergence of a neural network training phase that it has become a common practice. Our proposed DBN algorithm retains the overall structure of the original BN algorithm while introducing a weighted averaging update to some trainable parameters. We provide an analysis of the convergence of the DBN algorithm that converges to a stationary point with respect to the trainable parameters. Our analysis can be easily generalized to the original BN algorithm by setting some parameters to constant. To the best of our knowledge, this analysis is the first of its kind for convergence with BN. We analyze a two-layer model with arbitrary activation functions. Common activation functions, such as ReLU and any smooth activation functions, meet our assumptions. In the numerical experiments, we test the proposed algorithm on complex modern CNN models with stochastic gradients (SGs) and ReLU activation on regression, classification, and image reconstruction tasks. We observe that DBN outperforms the original BN algorithm and benchmark layer normalization (LN) on the MNIST, NI, CIFAR-10, CIFAR-100, and Caltech-UCSD Birds-200-2011 datasets with modern complex CNN models such as Resnet-18 and typical FNN models.

Quaternion fast and accurate polar harmonic Fourier moments for color image analysis and object recognition

Image moments, as a kind of global feature descriptor of images, have become a valuable tool for pattern recognition and image analysis. However, traditional methods are mainly used to deal with grayscale images. In this paper, we apply quaternions to fast and accurate polar harmonic Fourier moments, proposing a kind of quaternion fast and accurate polar harmonic Fourier moment (QFAPHFM) capable of handling color images. Furthermore, this paper provides a detailed analysis of the invariance of QFAPHFMs under rotation, scaling, and translation transformations. The experimental results show that QFAPHFMs exhibit excellent performance in both image reconstruction and object recognition tasks. QFAPHFMs achieve accurate image reconstruction under noiseless and noisy conditions, and demonstrate excellent recognition performance in the color-based object recognition tasks.

Learning Cross-modality Interaction for Robust Depth Perception of Autonomous Driving

As one of the fundamental tasks of autonomous driving, depth perception aims to perceive physical objects in three dimensions and to judge their distances away from the ego vehicle. Although great efforts have been made for depth perception, LiDAR-based and camera-based solutions have limitations with low accuracy and poor robustness for noise input. With the integration of monocular cameras and LiDAR sensors in autonomous vehicles, in this article, we introduce a two-stream architecture to learn the modality interaction representation under the guidance of an image reconstruction task to compensate for the deficiencies of each modality in a parallel manner. Specifically, in the two-stream architecture, the multi-scale cross-modality interactions are preserved via a cascading interaction network under the guidance of the reconstruction task. Next, the shared representation of modality interaction is integrated to infer the dense depth map due to the complementarity and heterogeneity of the two modalities. We evaluated the proposed solution on the KITTI dataset and CALAR synthetic dataset. Our experimental results show that learning the coupled interaction of modalities under the guidance of an auxiliary task can lead to significant performance improvements. Furthermore, our approach is competitive against the state-of-the-art models and robust against the noisy input. The source code is available at https://github.com/tonyFengye/Code/tree/master .

Sparse Bayesian Deep Learning for Cross Domain Medical Image Reconstruction

Cross domain medical image reconstruction aims to address the issue that deep learning models trained solely on one source dataset might not generalize effectively to unseen target datasets from different hospitals. Some recent methods achieve satisfactory reconstruction performance, but often at the expense of extensive parameters and time consumption. To strike a balance between cross domain image reconstruction quality and model computational efficiency, we propose a lightweight sparse Bayesian deep learning method. Notably, we apply a fixed-form variational Bayes (FFVB) approach to quantify pixel-wise uncertainty priors derived from degradation distribution of the source domain. Furthermore, by integrating the uncertainty prior into the posterior sampled through stochastic gradient Langevin dynamics (SGLD), we develop a training strategy that dynamically generates and optimizes the prior distribution on the network weights for each unseen domain. This strategy enhances generalizability and ensures robust reconstruction performance. When evaluated on medical image reconstruction tasks, our proposed approach demonstrates impressive performance across various previously unseen domains.

Exploring Novel Self-Supervised Learning Techniques for Image Reconstruction Tasks

Image reconstruction tasks, such as super-resolution, inpainting, and denoising, play a crucial role in various computer vision applications. Traditional methods often rely heavily on large labeled datasets for training, which can be costly and time-consuming to acquire. Self-supervised learning has emerged as a promising alternative, aiming to reduce this dependency by leveraging the inherent structures within the data itself. In this paper, we explore novel self-supervised learning techniques tailored specifically for image reconstruction tasks. We propose approaches that exploit the inherent relationships between low and high-resolution images, utilize context-aware information for inpainting, and incorporate generative adversarial networks for denoising. Through extensive experimentation, we demonstrate the efficacy of our methods in achieving competitive performance compared to supervised approaches while significantly reducing the need for labeled data. Our findings pave the way for more efficient and scalable solutions in image reconstruction, offering practical benefits across a wide range of applications

Triplet-constrained deep hashing for chest X-ray image retrieval in COVID-19 assessment

Radiology images of the chest, such as computer tomography scans and X-rays, have been prominently used in computer-aided COVID-19 analysis. Learning-based radiology image retrieval has attracted increasing attention recently, which generally involves image feature extraction and finding matches in extensive image databases based on query images. Many deep hashing methods have been developed for chest radiology image search due to the high efficiency of retrieval using hash codes. However, they often overlook the complex triple associations between images; that is, images belonging to the same category tend to share similar characteristics and vice versa. To this end, we develop a triplet-constrained deep hashing (TCDH) framework for chest radiology image retrieval to facilitate automated analysis of COVID-19. The TCDH consists of two phases, including (a) feature extraction and (b) image retrieval. For feature extraction, we have introduced a triplet constraint and an image reconstruction task to enhance discriminative ability of learned features, and these features are then converted into binary hash codes to capture semantic information. Specifically, the triplet constraint is designed to pull closer samples within the same category and push apart samples from different categories. Additionally, an auxiliary image reconstruction task is employed during feature extraction to help effectively capture anatomical structures of images. For image retrieval, we utilize learned hash codes to conduct searches for medical images. Extensive experiments on 30,386 chest X-ray images demonstrate the superiority of the proposed method over several state-of-the-art approaches in automated image search. The code is now available online.

Score-Based Generative Models for PET Image Reconstruction

Score-based generative models have demonstrated highly promising results for medical image reconstruction tasks in magnetic resonance imaging or computed tomography. However, their application to Positron Emission Tomography (PET) is still largely unexplored. PET image reconstruction involves a variety of challenges, including Poisson noise with high variance and a wide dynamic range. To address these challenges, we propose several PET-specific adaptations of score-based generative models. The proposed framework is developed for both 2D and 3D PET. In addition, we provide an extension to guided reconstruction using magnetic resonance images. We validate the approach through extensive 2D and 3D &lt;emph&gt;in-silico&lt;/emph&gt; experiments with a model trained on patient-realistic data without lesions, and evaluate on data without lesions as well as out-of-distribution data with lesions. This demonstrates the proposed method’s robustness and significant potential for improved PET reconstruction.

Generalizable Reconstruction for Accelerating MR Imaging via Federated Learning with Neural Architecture Search.

Heterogeneous data captured by different scanning devices and imaging protocols can affect the generalization performance of the deep learning magnetic resonance (MR) reconstruction model. While a centralized training model is effective in mitigating this problem, it raises concerns about privacy protection. Federated learning is a distributed training paradigm that can utilize multi-institutional data for collaborative training without sharing data. However, existing federated learning MR image reconstruction methods rely on models designed manually by experts, which are complex and computationally expensive, suffering from performance degradation when facing heterogeneous data distributions. In addition, these methods give inadequate consideration to fairness issues, namely ensuring that the model's training does not introduce bias towards any specific dataset's distribution. To this end, this paper proposes a generalizable federated neural architecture search framework for accelerating MR imaging (GAutoMRI). Specifically, automatic neural architecture search is investigated for effective and efficient neural network representation learning of MR images from different centers. Furthermore, we design a fairness adjustment approach that can enable the model to learn features fairly from inconsistent distributions of different devices and centers, and thus facilitate the model to generalize well to the unseen center. Extensive experiments show that our proposed GAutoMRI has better performances and generalization ability compared with seven state-of-the-art federated learning methods. Moreover, the GAutoMRI model is significantly more lightweight, making it an efficient choice for MR image reconstruction tasks. The code will be made available at https://github.com/ternencewu123/GAutoMRI.

Data-Driven Gradient Regularization for Quasi-Newton Optimization in Iterative Grating Interferometry CT Reconstruction.

Grating interferometry CT (GI-CT) is a promising technology that could play an important role in future breast cancer imaging. Thanks to its sensitivity to refraction and small-angle scattering, GI-CT could augment the diagnostic content of conventional absorption-based CT. However, reconstructing GI-CT tomographies is a complex task because of ill problem conditioning and high noise amplitudes. It has previously been shown that combining data-driven regularization with iterative reconstruction is promising for tackling challenging inverse problems in medical imaging. In this work, we present an algorithm that allows seamless combination of data-driven regularization with quasi-Newton solvers, which can better deal with ill-conditioned problems compared to gradient descent-based optimization algorithms. Contrary to most available algorithms, our method applies regularization in the gradient domain rather than in the image domain. This comes with a crucial advantage when applied in conjunction with quasi-Newton solvers: the Hessian is approximated solely based on denoised data. We apply the proposed method, which we call GradReg, to both conventional breast CT and GI-CT and show that both significantly benefit from our approach in terms of dose efficiency. Moreover, our results suggest that thanks to its sharper gradients that carry more high spatial-frequency content, GI-CT can benefit more from GradReg compared to conventional breast CT. Crucially, GradReg can be applied to any image reconstruction task which relies on gradient-based updates.

Twin-SegNet: Dynamically coupled complementary segmentation networks for generalized medical image segmentation

Image segmentation is deemed an important task in biomedicine, often required for proper diagnosis and prognosis of many diseases. Deep learning (DL) based segmentation methods have received considerable attention in recent years due to the increasing availability of clinical datasets. Many novel ideas have been proposed over the years driving progress in the field of automatic segmentation research. Contrary to the theme of contemporary literature, we demonstrate that considering the background tissue segmentation task alongside the main foreground task can improve overall segmentation performance when considered from a general medical image segmentation perspective. We, therefore, propose a DL framework called Twin Segmentation Network (Twin-SegNet) that ties together two streams (foreground and background) through an image reconstruction task. A boxed Mean Squared Error loss is proposed to complement the dice losses from both streams. We furthermore propose a Wavelet Convolutional Block (WCB) to enhance the edge information extracting capabilities of both streams and also a Partial Channel Recalibration (PCR) block to allow mutual feature exchange between the two streams so that each stream can emphasize more on channels with more discriminative and relevant features. We present experimental results on five public datasets: BUSI, GLAS, ISIC-2018, MoNuSeg, and CVC-ClinicDB. Unlike conventional baselines that demonstrate convincing performance in some datasets and poor performance in others, Twin-SegNet is able to consistently achieve state-of-the-art results with impressive F1 scores of 88.46%, 93.11%, 91.61%, 81.78% and 94.69% on each dataset respectively, showing its great potential as a general segmentation framework.

Image Reconstruction by VAE

Variational Autoencoders (VAEs) have become a recognized tool for learning data distributions in various applications. Building on previous research, this paper investigates image reconstruction by incorporating Planar Flows into VAEs. As a form of Normalizing Flows, Planar Flows potentially aid VAEs in modeling more intricate distributions, which may enhance the quality of reconstructed images. Our study employs a well-established architecture that integrates Planar Flows with VAEs. Experiments on MNIST are provided to compare our replication efforts with traditional VAEs. The results suggest that utilizing Planar Flows can offer certain improvements in image reconstruction. This work further solidifies the understanding of the combined use of VAEs and Planar Flows in image reconstruction tasks.