Reversible Diffusion Processes Research Articles

Multi-plane denoising diffusion-based dimensionality expansion for 2D-to-3D reconstruction of microstructures with harmonized sampling

Acquiring reliable microstructure datasets is a pivotal step toward the systematic design of materials with the aid of integrated computational materials engineering (ICME) approaches. However, obtaining three-dimensional (3D) microstructure datasets is often challenging due to high experimental costs or technical limitations, while acquiring two-dimensional (2D) micrographs is comparatively easier. To deal with this issue, this study proposes a novel framework called ‘Micro3Diff’ for 2D-to-3D reconstruction of microstructures using diffusion-based generative models (DGMs). Specifically, this approach solely requires pre-trained DGMs for the generation of 2D samples, and dimensionality expansion (2D-to-3D) takes place only during the generation process (i.e., reverse diffusion process). The proposed framework incorporates a concept referred to as ‘multi-plane denoising diffusion’, which transforms noisy samples (i.e., latent variables) from different planes into the data structure while maintaining spatial connectivity in 3D space. Furthermore, a harmonized sampling process is developed to address possible deviations from the reverse Markov chain of DGMs during the dimensionality expansion. Combined, we demonstrate the feasibility of Micro3Diff in reconstructing 3D samples with connected slices that maintain morphologically equivalence to the original 2D images. To validate the performance of Micro3Diff, various types of microstructures (synthetic or experimentally observed) are reconstructed, and the quality of the generated samples is assessed both qualitatively and quantitatively. The successful reconstruction outcomes inspire the potential utilization of Micro3Diff in upcoming ICME applications while achieving a breakthrough in comprehending and manipulating the latent space of DGMs.

Diff-IF: Multi-modality image fusion via diffusion model with fusion knowledge prior

Multi-modality image fusion (MMIF) aims to aggregate the complementary information from diverse source image domains. As the generative adversarial network-based methods have been the primary choice and demonstrated satisfactory fusion performance, they suffer from the unstable training and mode collapse. To tackle this challenge, we propose a novel diffusion model for MMIF incorporating fusion knowledge prior, termed as Diff-IF. Diff-IF proposes a trainable diffusion model paradigm for multi-modality image fusion, resolving the issue of lacking the ground truth for the diffusion model in image fusion tasks. It decomposes the diffusion-based image fusion method into conditional diffusion model and fusion knowledge prior with the targeted search to derive the prior distribution for the specific image fusion task. In particular, the forward diffusion process is guided by the fusion knowledge prior distribution through targeted search, while the reverse diffusion process is designed to generate high-quality fused images. Extensive experiments demonstrate that Diff-IF achieves outstanding performance, including exemplary visual preservation, and good preservation of weak textures, across various MMIF tasks such as infrared-visible image fusion and medical image fusion. The code will be available at https://github.com/XunpengYi/Diff-IF.

HeadDiff: Exploring Rotation Uncertainty With Diffusion Models for Head Pose Estimation.

In this paper, we propose a probabilistic regression diffusion model for head pose estimation, dubbed HeadDiff, which typically addresses the rotation uncertainty, especially when faces are captured in wild conditions. Unlike conventional image-to-pose methods which cannot explicitly establish the rotational manifold of head poses, our HeadDiff aims to ensure the pose rotation via the diffusion process and in parallel, refine the mapping process iteratively. Specifically, we initially formulate the head pose estimation problem as a reverse diffusion process, defining a paradigm for progressive denoising on the manifold, which explores the uncertainty by decomposing the large gap into intermediate steps. Moreover, our HeadDiff is equipped with an isotropic Gaussian distribution by encoding the incoherence information in our rotation representation. Finally, we learn the facial relationship of nearest neighbors with a cycle-consistent constraint for robust pose estimation versus diverse shape variations. Experimental results on multiple datasets demonstrate that our proposed method outperforms existing state-of-the-art techniques without auxiliary data.

Synthetic CT generation from MRI using 3D transformer-based denoising diffusion model.

Magnetic resonance imaging (MRI)-based synthetic computed tomography (sCT) simplifies radiation therapy treatment planning by eliminating the need for CT simulation and error-prone image registration, ultimately reducing patient radiation dose and setup uncertainty. In this work, we propose a MRI-to-CT transformer-based improved denoising diffusion probabilistic model (MC-IDDPM) to translate MRI into high-quality sCT to facilitate radiation treatment planning. MC-IDDPM implements diffusion processes with a shifted-window transformer network to generate sCT from MRI. The proposed model consists of two processes: a forward process, which involves adding Gaussian noise to real CT scans to create noisy images, and a reverse process, in which a shifted-window transformer V-net (Swin-Vnet) denoises the noisy CT scans conditioned on the MRI from the same patient to produce noise-free CT scans. With an optimally trained Swin-Vnet, the reverse diffusion process was used to generate noise-free sCT scans matching MRI anatomy. We evaluated the proposed method by generating sCT from MRI on an institutional brain dataset and an institutional prostate dataset. Quantitative evaluations were conducted using several metrics, including Mean Absolute Error (MAE), Peak Signal-to-Noise Ratio (PSNR), Multi-scale Structure Similarity Index (SSIM), and Normalized Cross Correlation (NCC). Dosimetry analyses were also performed, including comparisons of mean dose and target dose coverages for 95% and 99%. MC-IDDPM generated brain sCTs with state-of-the-art quantitative results with MAE 48.825±21.491 HU, PSNR 26.491±2.814dB, SSIM 0.947±0.032, and NCC 0.976±0.019. For the prostate dataset: MAE 55.124±9.414 HU, PSNR 28.708±2.112dB, SSIM 0.878±0.040, and NCC 0.940±0.039. MC-IDDPM demonstrates a statistically significant improvement (with p<0.05) in most metrics when compared to competing networks, for both brain and prostate synthetic CT. Dosimetry analyses indicated that the target dose coverage differences by using CT and sCT were within±0.34%. We have developed and validated a novel approach for generating CT images from routine MRIs using a transformer-based improved DDPM. This model effectively captures the complex relationship between CT and MRI images, allowing for robust and high-quality synthetic CT images to be generated in a matter of minutes. This approach has the potential to greatly simplify the treatment planning process for radiation therapy by eliminating the need for additional CT scans, reducing the amount of time patients spend in treatment planning, and enhancing the accuracy of treatment delivery.

Double reverse diffusion for realistic garment reconstruction from images

Creating realistic digital 3D avatars has been getting more attention thanks to the introduction of new multimedia formats such as augmented and virtual reality. An important factor making avatars realistic is clothes. In this paper, we investigate a new method to reconstruct realistic garments from a set of images and body information. Early methods working on realistic images struggle to faithfully reconstruct the garment details. As deep learning is increasingly applied to geometric data which can conveniently represent garments, we devise a novel deep learning-based solution to the garment reconstruction problem. We offer a new perspective on the reconstruction problem and treat it as a reversion of the smoothing diffusion process. To achieve this goal, we propose to deform the smoothed human mesh into a clothed human via a Double Reverse Diffusion (DReD) process. For the first reverse diffusion, we introduce a novel operator called Graph Long Short-Term Memory (GraphLSTM) which recursively diffuses features to produce a deformed mesh by modeling the relationships between vertices. Then, the output mesh can be repeatedly upsampled and deformed by the above pipeline to obtain finer garment details, which can be seen as another reverse diffusion process. To obtain features for the reverse diffusion, we extract pixel-aligned features transferred from images and explore to incorporate the visibility of garments from the image viewpoints. Through detailed experiments on two public datasets, we demonstrate that DReD synthesizes more realistic wrinkled garments with lower errors and offers faster inference than previous methods.

DiffuCom: A novel diffusion model for comment generation

Given that the diffusion model has good advantages in diversity, an increasing number of researchers have recently begun exploring natural language generation. However, despite significant progress in modeling based on text diffusion, there are still several challenges to be overcome in comment tasks. On the one hand, the source sentences in the comment task are long. If each timestep in the diffusion carries and computes the source sentence, it will significantly increase the computational overhead. On the other hand, the diffusion model is essentially an iterative non-autoregressive model, which lacks target decomposition in the decoding. This makes vanilla cross-attention prone to overlook local perceptual content, leading to the problem of poor correlation. To solve the aforementioned problem, we propose a novel diffusion model for comment generation (named DiffuCom). Specifically, DiffuCom employs the transformer-based encoder–decoder structure, encoding only once during the reverse diffusion process, effectively reducing computational overhead. Additionally, we design a context-aware cross-attention module (CACAM) that integrates global and local cross-attention information through a gating mechanism, which more accurately captures relevant fragment information in the source sentence and improves the correlation. Second, self-conditioning technology was introduced in the training process, which can effectively improve the quality of the samples. Furthermore, to enhance the controllability of DiffuCom, we also propose a blended control. It combines class-conditional and classifier guidance by utilizing the difference between training and inference. Extensive experiments on four comment datasets demonstrate that the proposed DiffuCom model can generate comments that are both diverse and relevant.

Enhancing Remote Sensing Image Super-Resolution with Efficient Hybrid Conditional Diffusion Model

Recently, optical remote-sensing images have been widely applied in fields such as environmental monitoring and land cover classification. However, due to limitations in imaging equipment and other factors, low-resolution images that are unfavorable for image analysis are often obtained. Although existing image super-resolution algorithms can enhance image resolution, these algorithms are not specifically designed for the characteristics of remote-sensing images and cannot effectively recover high-resolution images. Therefore, this paper proposes a novel remote-sensing image super-resolution algorithm based on an efficient hybrid conditional diffusion model (EHC-DMSR). The algorithm applies the theory of diffusion models to remote-sensing image super-resolution. Firstly, the comprehensive features of low-resolution images are extracted through a transformer network and CNN to serve as conditions for guiding image generation. Furthermore, to constrain the diffusion model and generate more high-frequency information, a Fourier high-frequency spatial constraint is proposed to emphasize high-frequency spatial loss and optimize the reverse diffusion direction. To address the time-consuming issue of the diffusion model during the reverse diffusion process, a feature-distillation-based method is proposed to reduce the computational load of U-Net, thereby shortening the inference time without affecting the super-resolution performance. Extensive experiments on multiple test datasets demonstrated that our proposed algorithm not only achieves excellent results in quantitative evaluation metrics but also generates sharper super-resolved images with rich detailed information.

Human Joint Kinematics Diffusion-Refinement for Stochastic Motion Prediction

Stochastic human motion prediction aims to forecast multiple plausible future motions given a single pose sequence from the past. Most previous works focus on designing elaborate losses to improve the accuracy, while the diversity is typically characterized by randomly sampling a set of latent variables from the latent prior, which is then decoded into possible motions. This joint training of sampling and decoding, however, suffers from posterior collapse as the learned latent variables tend to be ignored by a strong decoder, leading to limited diversity. Alternatively, inspired by the diffusion process in nonequilibrium thermodynamics, we propose MotionDiff, a diffusion probabilistic model to treat the kinematics of human joints as heated particles, which will diffuse from original states to a noise distribution. This process not only offers a natural way to obtain the "whitened'' latents without any trainable parameters, but also introduces a new noise in each diffusion step, both of which facilitate more diverse motions. Human motion prediction is then regarded as the reverse diffusion process that converts the noise distribution into realistic future motions conditioned on the observed sequence. Specifically, MotionDiff consists of two parts: a spatial-temporal transformer-based diffusion network to generate diverse yet plausible motions, and a flexible refinement network to further enable geometric losses and align with the ground truth. Experimental results on two datasets demonstrate that our model yields the competitive performance in terms of both diversity and accuracy.

Image embedding for denoising generative models

Denoising Diffusion models are gaining increasing popularity in the field of generative modeling for several reasons, including the simple and stable training, the excellent generative quality, and the solid probabilistic foundation. In this article, we address the problem of embedding an image into the latent space of Denoising Diffusion Models, that is finding a suitable “noisy” image whose denoising results in the original image. We particularly focus on Denoising Diffusion Implicit Models due to the deterministic nature of their reverse diffusion process. As a side result of our investigation, we gain a deeper insight into the structure of the latent space of diffusion models, opening interesting perspectives on its exploration, the definition of semantic trajectories, and the manipulation/conditioning of encodings for editing purposes. A particularly interesting property highlighted by our research, which is also characteristic of this class of generative models, is the independence of the latent representation from the networks implementing the reverse diffusion process. In other words, a common seed passed to different networks (each trained on the same dataset), eventually results in identical images.

Stability of the Poincaré constant

We study stability of the sharp Poincaré constant of the invariant probability measure of a reversible diffusion process satisfying some natural conditions. The proof is based on the spectral interpretation of Poincaré inequalities and Stein’s method. In particular, these results are applied to gamma distributions, to the Brownian motion on spheres and to the Brascamp-Lieb inequality for one-dimensional log-concave measures.

How Much Is Enough? A Study on Diffusion Times in Score-Based Generative Models

Score-based diffusion models are a class of generative models whose dynamics is described by stochastic differential equations that map noise into data. While recent works have started to lay down a theoretical foundation for these models, a detailed understanding of the role of the diffusion time T is still lacking. Current best practice advocates for a large T to ensure that the forward dynamics brings the diffusion sufficiently close to a known and simple noise distribution; however, a smaller value of T should be preferred for a better approximation of the score-matching objective and higher computational efficiency. Starting from a variational interpretation of diffusion models, in this work we quantify this trade-off and suggest a new method to improve quality and efficiency of both training and sampling, by adopting smaller diffusion times. Indeed, we show how an auxiliary model can be used to bridge the gap between the ideal and the simulated forward dynamics, followed by a standard reverse diffusion process. Empirical results support our analysis; for image data, our method is competitive with regard to the state of the art, according to standard sample quality metrics and log-likelihood.

Dif-Fusion: Towards High Color Fidelity in Infrared and Visible Image Fusion with Diffusion Models.

Color plays an important role in human visual perception, reflecting the spectrum of objects. However, the existing infrared and visible image fusion methods rarely explore how to handle multi-spectral/channel data directly and achieve high color fidelity. This paper addresses the above issue by proposing a novel method with diffusion models, termed as Dif-Fusion, to generate the distribution of the multi-channel input data, which increases the ability of multi-source information aggregation and the fidelity of colors. In specific, instead of converting multi-channel images into single-channel data in existing fusion methods, we create the multi-channel data distribution with a denoising network in a latent space with forward and reverse diffusion process. Then, we use the the denoising network to extract the multi-channel diffusion features with both visible and infrared information. Finally, we feed the multi-channel diffusion features to the multi-channel fusion module to directly generate the three-channel fused image. To retain the texture and intensity information, we propose multi-channel gradient loss and intensity loss. Along with the current evaluation metrics for measuring texture and intensity fidelity, we introduce Delta E as a new evaluation metric to quantify color fidelity. Extensive experiments indicate that our method is more effective than other state-of-the-art image fusion methods, especially in color fidelity. The source code is available at https://github.com/GeoVectorMatrix/Dif-Fusion.

Adaptive invariant density estimation for continuous-time mixing Markov processes under sup-norm risk

Up to now, the nonparametric analysis of multidimensional continuous-time Markov processes has focussed strongly on specific model choices, mostly related to symmetry of the semigroup. While this approach allows to study the performance of estimators for the characteristics of the process in the minimax sense, it restricts the applicability of results to a rather constrained set of stochastic processes and in particular hardly allows incorporating jump structures. As a consequence, for many models of applied and theoretical interest, no statement can be made about the robustness of typical statistical procedures beyond the beautiful, but limited framework available in the literature. To close this gap, we identify $\beta$-mixing of the process and heat kernel bounds on the transition density as a suitable combination to obtain $\sup$-norm and $L^2$ kernel invariant density estimation rates matching the case of reversible multidimenisonal diffusion processes and outperforming density estimation based on discrete i.i.d. or weakly dependent data. Moreover, we demonstrate how up to $\log$-terms, optimal $\sup$-norm adaptive invariant density estimation can be achieved within our general framework based on tight uniform moment bounds and deviation inequalities for empirical processes associated to additive functionals of Markov processes. The underlying assumptions are verifiable with classical tools from stability theory of continuous time Markov processes and PDE techniques, which opens the door to evaluate statistical performance for a vast amount of Markov models. We highlight this point by showing how multidimensional jump SDEs with L\'evy driven jump part under different coefficient assumptions can be seamlessly integrated into our framework, thus establishing novel adaptive $\sup$-norm estimation rates for this class of processes.

Stability of higher order eigenvalues in dimension one

We study stability of the eigenvalues of the generator of a one dimensional reversible diffusion process satisfying some natural conditions. The proof is based on Stein’s method. In particular, these results are applied to the Normal distribution (via the Ornstein–Uhlenbeck process), to Gamma distributions (via the Laguerre process) and to Beta distributions (via the Jacobi process).

Variational Principles for Asymptotic Variance of General Markov Processes

A variational formula for the asymptotic variance of general Markov processes is obtained. As application, we get an upper bound of the mean exit time of reversible Markov processes, and some comparison theorems between the reversible and non-reversible diffusion processes.

Non-reversible Metastable Diffusions with Gibbs Invariant Measure II: Markov Chain Convergence

This article considers a class of metastable non-reversible diffusion processes whose invariant measure is a Gibbs measure associated with a Morse potential. In a companion paper (Lee and Seo in Probab Theory Relat Fields 182:849–903, 2022), we proved the Eyring–Kramers formula for the corresponding class of metastable diffusion processes. In this article, we further develop this result by proving that a suitably time-rescaled metastable diffusion process converges to a Markov chain on the deepest metastable valleys. This article is also an extension of (Rezakhanlou and Seo in https://arxiv.org/abs/1812.02069, 2018), which considered the same problem for metastable reversible diffusion processes. Our proof is based on the recently developed resolvent approach to metastability.

One Dimensional Critical Kinetic Fokker–Planck Equations, Bessel and Stable Processes

We consider a particle moving in one dimension, its velocity being a reversible diffusion process, with constant diffusion coefficient, of which the invariant measure behaves like $$(1+|v|)^{-\beta }$$ for some $$\beta >0$$ . We prove that, under a suitable rescaling, the position process resembles a Brownian motion if $$\beta \ge 5$$ , a stable process if $$\beta \in [1,5)$$ and an integrated symmetric Bessel process if $$\beta \in (0,1)$$ . The critical cases $$\beta =1$$ and $$\beta =5$$ require special rescalings. We recover some results of Nasreddine and Puel (ESAIM Math Model Numer Anal 49:1–17, 2015), Cattiaux et al. (Kinet Relat Models, to appear), Lebeau and Puel (Commun Math Phys, to appear. arXiv:1711.03060 ) and Barkai et al. (Phys Rev X 4:021036, 2014) on the kinetic Fokker–Planck equation, with an alternative approach.

Spectral thresholding for the estimation of Markov chain transition operators

We consider nonparametric estimation of the transition operator $P$ of a Markov chain and its transition density $p$ where the singular values of $P$ are assumed to decay exponentially fast. This is for instance the case for periodised, reversible multi-dimensional diffusion processes observed in low frequency. We investigate the performance of a spectral hard thresholded Galerkin-type estimator for $P$ and ${p}$, discarding most of the estimated singular triples. The construction is based on smooth basis functions such as wavelets or B-splines. We show its statistical optimality by establishing matching minimax upper and lower bounds in $L^2$-loss. Particularly, the effect of the dimensionality $d$ of the state space on the nonparametric rate improves from $2d$ to $d$ compared to the case without singular value decay.

Reversible Al Propagation in Si x Ge1-x Nanowires: Implications for Electrical Contact Formation.

While reversibility is a fundamental concept in thermodynamics, most reactions are not readily reversible, especially in solid-state physics. For example, thermal diffusion is a widely known concept, used among others to inject dopants into the substitutional positions in the matrix and improve device properties. Typically, such a diffusion process will create a concentration gradient extending over increasingly large regions, without possibility to reverse this effect. On the other hand, while the bottom-up growth of semiconducting nanowires is interesting, it can still be difficult to fabricate axial heterostructures with high control. In this paper, we report a thermally assisted partially reversible thermal diffusion process occurring in the solid-state reaction between an Al metal pad and a SixGe1–x alloy nanowire observed by in situ transmission electron microscopy. The thermally assisted reaction results in the creation of a Si-rich region sandwiched between the reacted Al and unreacted SixGe1–x part, forming an axial Al/Si/SixGe1–x heterostructure. Upon heating or (slow) cooling, the Al metal can repeatably move in and out of the SixGe1–x alloy nanowire while maintaining the rodlike geometry and crystallinity, allowing to fabricate and contact nanowire heterostructures in a reversible way in a single process step, compatible with current Si-based technology. This interesting system is promising for various applications, such as phase change memories in an all crystalline system with integrated contacts as well as Si/SixGe1–x/Si heterostructures for near-infrared sensing applications.

Anomalous diffusion for multi-dimensional critical kinetic Fokker–Planck equations

We consider a particle moving in $d\geq2$ dimensions, its velocity being a reversible diffusion process, with identity diffusion coefficient, of which the invariant measure behaves, roughly, like $(1+|v|)^{-\beta}$ as $|v|\to\infty$, for some constant $\beta>0$. We prove that for large times, after a suitable rescaling, the position process resembles a Brownian motion if $\beta\geq4+d$, a stable process if $\beta\in[d,4+d)$ and an integrated multi-dimensional generalization of a Bessel process if $\beta\in(d-2,d)$. The critical cases $\beta=d$, $\beta=1+d$ and $\beta=4+d$ require special rescalings.