Reversible Diffusion Processes Research Articles

Density‐Aware Diffusion Model for Efficient Image Dehazing

AbstractExisting image dehazing methods have made remarkable progress. However, they generally perform poorly on images with dense haze, and often suffer from unsatisfactory results with detail degradation or color distortion. In this paper, we propose a density‐aware diffusion model (DADM) for image dehazing. Guided by the haze density, our DADM can handle images with dense haze and complex environments. Specifically, we introduce a density‐aware dehazing network (DADNet) in the reverse diffusion process, which can help DADM gradually recover a clear haze‐free image from a haze image. To improve the performance of the network, we design a cross‐feature density extraction module (CDEModule) to extract the haze density for the image and a density‐guided feature fusion block (DFFBlock) to learn the effective contextual features. Furthermore, we introduce an indirect sampling strategy in the test sampling process, which not only suppresses the accumulation of errors but also ensures the stability of the results. Extensive experiments on popular benchmarks validate the superior performance of the proposed method. The code is released in https://github.com/benchacha/DADM.

MWRD (Mamba Wavelet Reverse Diffusion)—An Efficient Fundus Image Enhancement Network Based on an Improved State-Space Model

This study introduces an efficient fundus image enhancement framework based on an improved Mamba model and the Denoising Diffusion Probabilistic Model (DDPM). By integrating wavelet transform for local feature extraction and applying a reverse diffusion process, this approach significantly improves the effectiveness and efficiency of enhancing low-quality fundus images. The model achieves high-precision enhancement of retinal vessel details while substantially reducing the number of parameters. Comprehensive tests on publicly available datasets show that the suggested approach surpasses various advanced low-light image enhancement methods in both quantitative and qualitative assessments.

DM-HAP: Diffusion model for accurate hand pose prediction

Forecasting hand poses is a challenging task due to inherent uncertainties, occlusions, and inaccuracies in 3D pose estimation. Diffusion models provide a promising direction for predicting precise 3D hand poses under noisy conditions. In this work, we introduce Dual-diffusion, an innovative framework for the precise prediction of future hand poses. Our approach leverages the strengths of diffusion models by framing hand pose forecasting as a reverse diffusion process, effectively addressing the complexities of noisy hand joint movements and their subtleties. To enhance the learning of hand pose representations, we propose a unique neural architecture that simultaneously captures both local and global features. This is achieved through the deployment of Global and Local Diffusion (GLD) blocks within our network, which facilitate the exchange of information between local and global features. This diffusion-based interaction enables the integration of global hand actions and local finger actions, leading to a more powerful representation learning approach. We evaluate the effectiveness of our Dual-diffusion method on three public 3D hand pose estimation datasets (MSRA, F-PHAB, and BigHand2.2M), and our approach outperforms previous methods.

SIMTSeg: A self-supervised multivariate time series segmentation method with periodic subspace projection and reverse diffusion for industrial process

Subsequences with varied regimes in the industrial multivariate time series (MTS) are closely associated with the dynamic status of the multi-phased industrial process. Time series segmentation (TSS) provides insights into the underlying behavior of industrial systems. However, the complexity of industrial data poses significant challenges to the conventional TSS methods. Motivated by this, a Self-supervised Industrial Multivariate Time-series Segmentation method (SIMTSeg) is presented in this work. An MTS folding module based on Ramanujan periodic subspace projection is first proposed, where the MTS is reshaped into the 3D feature map to realize the compact representation of the intricate data dependencies. Subsequently, a self-supervised module based on the encoder-decoder architecture is adopted to address the problem of deficient and task-specific annotations in industrial data. The folded feature map is denoised step by step following the reverse diffusion process, and finally turns into the segmentation mask without redundant details. The proposed SIMTSeg has been validated by a popular industrial benchmark, the Tennessee Eastman Process, and outperforms the unsupervised data-driven baselines in terms of various performance metrics. SIMTSeg has no prerequisite on the number of segmentation points or regime types, and is capable of giving more meaningful segmentation results that are in line with the high-level semantics.

Physics-Informed DeepMRI: k-Space Interpolation Meets Heat Diffusion.

Recently, diffusion models have shown considerable promise for MRI reconstruction. However, extensive experimentation has revealed that these models are prone to generating artifacts due to the inherent randomness involved in generating images from pure noise. To achieve more controlled image reconstruction, we reexamine the concept of interpolatable physical priors in k-space data, focusing specifically on the interpolation of high-frequency (HF) k-space data from low-frequency (LF) k-space data. Broadly, this insight drives a shift in the generation paradigm from random noise to a more deterministic approach grounded in the existing LF k-space data. Building on this, we first establish a relationship between the interpolation of HF k-space data from LF k-space data and the reverse heat diffusion process, providing a fundamental framework for designing diffusion models that generate missing HF data. To further improve reconstruction accuracy, we integrate a traditional physics-informed k-space interpolation model into our diffusion framework as a data fidelity term. Experimental validation using publicly available datasets demonstrates that our approach significantly surpasses traditional k-space interpolation methods, deep learning-based k-space interpolation techniques, and conventional diffusion models, particularly in HF regions. Finally, we assess the generalization performance of our model across various out-of-distribution datasets. Our code are available at https://github.com/ZhuoxuCui/Heat-Diffusion.

ICDDPM: Image-conditioned denoising diffusion probabilistic model for real-world complex point cloud single view reconstruction

Point cloud single-view reconstruction (PC-SVR) generates high-quality point clouds from low-cost 2D images, offering a cost-effective solution to the expensive and inefficient acquisition of point cloud for real-world scene typically achieved through expensive LiDAR systems. However, current models that generate point clouds from real-world 2D images (e.g. maritime vessels) still have shortcomings in terms of quality and model generalization. In this paper, we proposed a image-conditioned denoising diffusion probabilistic model (ICDDPM) for real-world complex point cloud single view reconstruction to address these issues. We re-designed the structure of classic diffusion model, use latent shape vectors to seamlessly integrate 2D image encoder, point cloud encoder, and conditional diffusion model, to cater PC-SVR task. By guiding the diffusion process with the 2D images, which serve as crucial conditional information, ICDDPM achieves end-to-end point cloud generation with superior quality. 2D images are also employed as input in the reverse diffusion process to further achieve point cloud generation. We conducted qualitative and quantitative experiments on synthetic dataset ShapeNet and real-world dataset PASCAL3D+ (focused on experiments of vessel point cloud data specifically). The results indicate that the ICDDPM model demonstrates superior performance compared to state-of-the-art models. It is capable of generating point clouds with a greater level of global and local details from various 2D image data. Additionally, the model exhibits strong generalization abilities and requires fewer computational resources.

Non-reversible lifts of reversible diffusion processes and relaxation times

AbstractWe propose a new concept of lifts of reversible diffusion processes and show that various well-known non-reversible Markov processes arising in applications are lifts in this sense of simple reversible diffusions. Furthermore, we introduce a concept of non-asymptotic relaxation times and show that these can at most be reduced by a square root through lifting, generalising a related result in discrete time. Finally, we demonstrate how the recently developed approach to quantitative hypocoercivity based on space–time Poincaré inequalities can be rephrased and simplified in the language of lifts and how it can be applied to find optimal lifts.

Diffusing on Two Levels and Optimizing for Multiple Properties: A Novel Approach to Generating Molecules with Desirable Properties.

In the past decade, Artificial Intelligence (AI) driven drug design and discovery has been a hot research topic in the AI area, where an important branch is molecule generation by generative models, from GAN-based models and VAE-based models to the latest diffusion-based models. However, most existing models pursue mainly the basic properties like validity and uniqueness of the generated molecules, a few go further to explicitly optimize one single important molecular property (e.g., QED or PlogP), which makes most generated molecules little usefulness in practice. In this paper, we present a novel approach to generating molecules with desirable properties, which expands the diffusion model framework with multiple innovative designs. The novelty is two-fold. On the one hand, considering that the structures of molecules are complex and diverse, and molecular properties are usually determined by some substructures (e.g., pharmacophores), we propose to perform diffusion on two structural levels: molecules and molecular fragments respectively, with which a mixed Gaussian distribution is obtained for the reverse diffusion process. To get desirable molecular fragments, we develop a novel electronic effect based fragmentation method. On the other hand, we introduce two ways to explicitly optimize multiple molecular properties under the diffusion model framework. First, as potential drug molecules must be chemically valid, we optimize molecular validity by an energy-guidance function. Second, since potential drug molecules should be desirable in various properties, we employ a multi-objective mechanism to optimize multiple molecular properties simultaneously. Extensive experiments with two benchmark datasets QM9 and ZINC250k show that the molecules generated by our proposed method have better validity, uniqueness, novelty, Fr´echet ChemNet Distance (FCD), QED, and PlogP than those generated by current SOTA models. The Code of D2L-OMP is available at https://github.com/bz99bz/D2L-OMP.

Joint diffusion: mutual consistency-driven diffusion model for PET-MRI co-reconstruction

Objective. Positron Emission Tomography and Magnetic Resonance Imaging (PET-MRI) systems can obtain functional and anatomical scans. But PET suffers from a low signal-to-noise ratio, while MRI are time-consuming. To address time-consuming, an effective strategy involves reducing k-space data collection, albeit at the cost of lowering image quality. This study aims to leverage the inherent complementarity within PET-MRI data to enhance the image quality of PET-MRI. Approach. A novel PET-MRI joint reconstruction model, termed MC-Diffusion, is proposed in the Bayesian framework. The joint reconstruction problem is transformed into a joint regularization problem, where data fidelity terms of PET and MRI are expressed independently. The regular term, the derivative of the logarithm of the joint probability distribution of PET and MRI, employs a joint score-based diffusion model for learning. The diffusion model involves the forward diffusion process and the reverse diffusion process. The forward diffusion process adds noise to transform a complex joint data distribution into a known joint prior distribution for PET and MRI simultaneously, resembling a denoiser. The reverse diffusion process removes noise using a denoiser to revert the joint prior distribution to the original joint data distribution, effectively utilizing joint probability distribution to describe the correlations of PET and MRI for improved quality of joint reconstruction. Main results. Qualitative and quantitative improvements are observed with the MC-Diffusion model. Comparative analysis against LPLS and Joint ISAT-net on the ADNI dataset demonstrates superior performance by exploiting complementary information between PET and MRI. The MC-Diffusion model effectively enhances the quality of PET and MRI images. Significance. This study employs the MC-Diffusion model to enhance the quality of PET-MRI images by integrating the fundamental principles of PET and MRI modalities and leveraging their inherent complementarity. Furthermore, utilizing the diffusion model to learn the joint probability distribution of PET and MRI, thereby elucidating their latent correlation, facilitates a more profound comprehension of the priors obtained through deep learning, contrasting with black-box prior or artificially constructed structural similarities.

A supersymmetric quantum perspective on the explicit large deviations for reversible Markov jump processes, with applications to pure and random spin chains

The large deviations at various levels that are explicit for Markov jump processes satisfying detailed balance are revisited in terms of the supersymmetric quantum Hamiltonian H that can be obtained from the Markov generator via a similarity transformation. We first focus on the large deviations at level 2 for the empirical density p^(C) of the configurations C seen during a trajectory over the large time window [0,T] , and rewrite the explicit Donsker–Varadhan rate function as the matrix element I[2][p^(.)]=⟨p^|H|p^⟩ involving the square-root ket |p^⟩ . (The analog formula is also discussed for reversible diffusion processes as a comparison.) We then consider the explicit rate functions at higher levels, in particular for the joint probability of the empirical density p^(C) and the empirical local activities a^(C,C′) characterizing the density of jumps between two configurations (C,C′) . Finally, the explicit rate function for the joint probability of the empirical density p^(C) and of the empirical total activity A^ that represents the total density of jumps of a long trajectory is written in terms of the two matrix elements ⟨p^|H|p^⟩ and ⟨p^|Hoff|p^⟩ , where Hoff represents the off-diagonal part of the supersymmetric Hamiltonian H. This general framework is then applied to pure or random spin chains with single-spin-flip or two-spin-flip transition rates, where the supersymmetric Hamiltonian H corresponds to quantum spin chains with local interactions involving Pauli matrices of two or three neighboring sites. It is then useful to introduce the quantum density matrix ρ^=|p^⟩⟨p^| associated with the empirical density p^(.) in order to rewrite the various rate functions in terms of reduced density matrices involving only two or three neighboring sites.

Synthesizing PET images from high-field and ultra-high-field MR images using joint diffusion attention model.

Magnetic Resonance Imaging (MRI) and Positron Emission Tomography (PET) stand as pivotal diagnostic tools for brain disorders, offering the potential for mutually enriching disease diagnostic perspectives. However, the costs associated with PET scans and the inherent radioactivity have limited the widespread application of PET. Furthermore, it is noteworthy to highlight the promising potential of high-field and ultra-high-field neuroimaging in cognitive neuroscience research and clinical practice. With the enhancement of MRI resolution, a related question arises: can high-resolution MRI improve the quality of PETimages? This study aims to enhance the quality of synthesized PET images by leveraging the superior resolution capabilities provided by high-field and ultra-high-fieldMRI. From a statistical perspective, the joint probability distribution is considered the most direct and fundamental approach for representing the correlation between PET and MRI. In this study, we proposed a novel model, the joint diffusion attention model, namely, the joint diffusion attention model (JDAM), which primarily focuses on learning information about the joint probability distribution. JDAM consists of two primary processes: the diffusion process and the sampling process. During the diffusion process, PET gradually transforms into a Gaussian noise distribution by adding Gaussian noise, while MRI remains fixed. The central objective of the diffusion process is to learn the gradient of the logarithm of the joint probability distribution between MRI and noise PET. The sampling process operates as a predictor-corrector. The predictor initiates a reverse diffusion process, and the corrector applies Langevindynamics. Experimental results from the publicly available Alzheimer's Disease Neuroimaging Initiative dataset highlight the effectiveness of the proposed model compared to state-of-the-art (SOTA) models such as Pix2pix and CycleGAN. Significantly, synthetic PET images guided by ultra-high-field MRI exhibit marked improvements in signal-to-noise characteristics when contrasted with those generated from high-field MRI data. These results have been endorsed by medical experts, who consider the PET images synthesized through JDAM to possess scientific merit. This endorsement is based on their symmetrical features and precise representation of regions displaying hypometabolism, a hallmark of Alzheimer'sdisease. This study establishes the feasibility of generating PET images from MRI. Synthesis of PET by JDAM significantly enhances image quality compared to SOTAmodels.

Multi-plane denoising diffusion-based dimensionality expansion for 2D-to-3D reconstruction of microstructures with harmonized sampling

Acquiring reliable microstructure datasets is a pivotal step toward the systematic design of materials with the aid of integrated computational materials engineering (ICME) approaches. However, obtaining three-dimensional (3D) microstructure datasets is often challenging due to high experimental costs or technical limitations, while acquiring two-dimensional (2D) micrographs is comparatively easier. To deal with this issue, this study proposes a novel framework called ‘Micro3Diff’ for 2D-to-3D reconstruction of microstructures using diffusion-based generative models (DGMs). Specifically, this approach solely requires pre-trained DGMs for the generation of 2D samples, and dimensionality expansion (2D-to-3D) takes place only during the generation process (i.e., reverse diffusion process). The proposed framework incorporates a concept referred to as ‘multi-plane denoising diffusion’, which transforms noisy samples (i.e., latent variables) from different planes into the data structure while maintaining spatial connectivity in 3D space. Furthermore, a harmonized sampling process is developed to address possible deviations from the reverse Markov chain of DGMs during the dimensionality expansion. Combined, we demonstrate the feasibility of Micro3Diff in reconstructing 3D samples with connected slices that maintain morphologically equivalence to the original 2D images. To validate the performance of Micro3Diff, various types of microstructures (synthetic or experimentally observed) are reconstructed, and the quality of the generated samples is assessed both qualitatively and quantitatively. The successful reconstruction outcomes inspire the potential utilization of Micro3Diff in upcoming ICME applications while achieving a breakthrough in comprehending and manipulating the latent space of DGMs.

Diff-IF: Multi-modality image fusion via diffusion model with fusion knowledge prior

Multi-modality image fusion (MMIF) aims to aggregate the complementary information from diverse source image domains. As the generative adversarial network-based methods have been the primary choice and demonstrated satisfactory fusion performance, they suffer from the unstable training and mode collapse. To tackle this challenge, we propose a novel diffusion model for MMIF incorporating fusion knowledge prior, termed as Diff-IF. Diff-IF proposes a trainable diffusion model paradigm for multi-modality image fusion, resolving the issue of lacking the ground truth for the diffusion model in image fusion tasks. It decomposes the diffusion-based image fusion method into conditional diffusion model and fusion knowledge prior with the targeted search to derive the prior distribution for the specific image fusion task. In particular, the forward diffusion process is guided by the fusion knowledge prior distribution through targeted search, while the reverse diffusion process is designed to generate high-quality fused images. Extensive experiments demonstrate that Diff-IF achieves outstanding performance, including exemplary visual preservation, and good preservation of weak textures, across various MMIF tasks such as infrared-visible image fusion and medical image fusion. The code will be available at https://github.com/XunpengYi/Diff-IF.

HeadDiff: Exploring Rotation Uncertainty With Diffusion Models for Head Pose Estimation.

In this paper, we propose a probabilistic regression diffusion model for head pose estimation, dubbed HeadDiff, which typically addresses the rotation uncertainty, especially when faces are captured in wild conditions. Unlike conventional image-to-pose methods which cannot explicitly establish the rotational manifold of head poses, our HeadDiff aims to ensure the pose rotation via the diffusion process and in parallel, refine the mapping process iteratively. Specifically, we initially formulate the head pose estimation problem as a reverse diffusion process, defining a paradigm for progressive denoising on the manifold, which explores the uncertainty by decomposing the large gap into intermediate steps. Moreover, our HeadDiff is equipped with an isotropic Gaussian distribution by encoding the incoherence information in our rotation representation. Finally, we learn the facial relationship of nearest neighbors with a cycle-consistent constraint for robust pose estimation versus diverse shape variations. Experimental results on multiple datasets demonstrate that our proposed method outperforms existing state-of-the-art techniques without auxiliary data.

Synthetic CT generation from MRI using 3D transformer-based denoising diffusion model.

Magnetic resonance imaging (MRI)-based synthetic computed tomography (sCT) simplifies radiation therapy treatment planning by eliminating the need for CT simulation and error-prone image registration, ultimately reducing patient radiation dose and setup uncertainty. In this work, we propose a MRI-to-CT transformer-based improved denoising diffusion probabilistic model (MC-IDDPM) to translate MRI into high-quality sCT to facilitate radiation treatment planning. MC-IDDPM implements diffusion processes with a shifted-window transformer network to generate sCT from MRI. The proposed model consists of two processes: a forward process, which involves adding Gaussian noise to real CT scans to create noisy images, and a reverse process, in which a shifted-window transformer V-net (Swin-Vnet) denoises the noisy CT scans conditioned on the MRI from the same patient to produce noise-free CT scans. With an optimally trained Swin-Vnet, the reverse diffusion process was used to generate noise-free sCT scans matching MRI anatomy. We evaluated the proposed method by generating sCT from MRI on an institutional brain dataset and an institutional prostate dataset. Quantitative evaluations were conducted using several metrics, including Mean Absolute Error (MAE), Peak Signal-to-Noise Ratio (PSNR), Multi-scale Structure Similarity Index (SSIM), and Normalized Cross Correlation (NCC). Dosimetry analyses were also performed, including comparisons of mean dose and target dose coverages for 95% and 99%. MC-IDDPM generated brain sCTs with state-of-the-art quantitative results with MAE 48.825±21.491 HU, PSNR 26.491±2.814dB, SSIM 0.947±0.032, and NCC 0.976±0.019. For the prostate dataset: MAE 55.124±9.414 HU, PSNR 28.708±2.112dB, SSIM 0.878±0.040, and NCC 0.940±0.039. MC-IDDPM demonstrates a statistically significant improvement (with p<0.05) in most metrics when compared to competing networks, for both brain and prostate synthetic CT. Dosimetry analyses indicated that the target dose coverage differences by using CT and sCT were within±0.34%. We have developed and validated a novel approach for generating CT images from routine MRIs using a transformer-based improved DDPM. This model effectively captures the complex relationship between CT and MRI images, allowing for robust and high-quality synthetic CT images to be generated in a matter of minutes. This approach has the potential to greatly simplify the treatment planning process for radiation therapy by eliminating the need for additional CT scans, reducing the amount of time patients spend in treatment planning, and enhancing the accuracy of treatment delivery.

Double reverse diffusion for realistic garment reconstruction from images

Creating realistic digital 3D avatars has been getting more attention thanks to the introduction of new multimedia formats such as augmented and virtual reality. An important factor making avatars realistic is clothes. In this paper, we investigate a new method to reconstruct realistic garments from a set of images and body information. Early methods working on realistic images struggle to faithfully reconstruct the garment details. As deep learning is increasingly applied to geometric data which can conveniently represent garments, we devise a novel deep learning-based solution to the garment reconstruction problem. We offer a new perspective on the reconstruction problem and treat it as a reversion of the smoothing diffusion process. To achieve this goal, we propose to deform the smoothed human mesh into a clothed human via a Double Reverse Diffusion (DReD) process. For the first reverse diffusion, we introduce a novel operator called Graph Long Short-Term Memory (GraphLSTM) which recursively diffuses features to produce a deformed mesh by modeling the relationships between vertices. Then, the output mesh can be repeatedly upsampled and deformed by the above pipeline to obtain finer garment details, which can be seen as another reverse diffusion process. To obtain features for the reverse diffusion, we extract pixel-aligned features transferred from images and explore to incorporate the visibility of garments from the image viewpoints. Through detailed experiments on two public datasets, we demonstrate that DReD synthesizes more realistic wrinkled garments with lower errors and offers faster inference than previous methods.

DiffuCom: A novel diffusion model for comment generation

Given that the diffusion model has good advantages in diversity, an increasing number of researchers have recently begun exploring natural language generation. However, despite significant progress in modeling based on text diffusion, there are still several challenges to be overcome in comment tasks. On the one hand, the source sentences in the comment task are long. If each timestep in the diffusion carries and computes the source sentence, it will significantly increase the computational overhead. On the other hand, the diffusion model is essentially an iterative non-autoregressive model, which lacks target decomposition in the decoding. This makes vanilla cross-attention prone to overlook local perceptual content, leading to the problem of poor correlation. To solve the aforementioned problem, we propose a novel diffusion model for comment generation (named DiffuCom). Specifically, DiffuCom employs the transformer-based encoder–decoder structure, encoding only once during the reverse diffusion process, effectively reducing computational overhead. Additionally, we design a context-aware cross-attention module (CACAM) that integrates global and local cross-attention information through a gating mechanism, which more accurately captures relevant fragment information in the source sentence and improves the correlation. Second, self-conditioning technology was introduced in the training process, which can effectively improve the quality of the samples. Furthermore, to enhance the controllability of DiffuCom, we also propose a blended control. It combines class-conditional and classifier guidance by utilizing the difference between training and inference. Extensive experiments on four comment datasets demonstrate that the proposed DiffuCom model can generate comments that are both diverse and relevant.

Enhancing Remote Sensing Image Super-Resolution with Efficient Hybrid Conditional Diffusion Model

Recently, optical remote-sensing images have been widely applied in fields such as environmental monitoring and land cover classification. However, due to limitations in imaging equipment and other factors, low-resolution images that are unfavorable for image analysis are often obtained. Although existing image super-resolution algorithms can enhance image resolution, these algorithms are not specifically designed for the characteristics of remote-sensing images and cannot effectively recover high-resolution images. Therefore, this paper proposes a novel remote-sensing image super-resolution algorithm based on an efficient hybrid conditional diffusion model (EHC-DMSR). The algorithm applies the theory of diffusion models to remote-sensing image super-resolution. Firstly, the comprehensive features of low-resolution images are extracted through a transformer network and CNN to serve as conditions for guiding image generation. Furthermore, to constrain the diffusion model and generate more high-frequency information, a Fourier high-frequency spatial constraint is proposed to emphasize high-frequency spatial loss and optimize the reverse diffusion direction. To address the time-consuming issue of the diffusion model during the reverse diffusion process, a feature-distillation-based method is proposed to reduce the computational load of U-Net, thereby shortening the inference time without affecting the super-resolution performance. Extensive experiments on multiple test datasets demonstrated that our proposed algorithm not only achieves excellent results in quantitative evaluation metrics but also generates sharper super-resolved images with rich detailed information.

Human Joint Kinematics Diffusion-Refinement for Stochastic Motion Prediction

Stochastic human motion prediction aims to forecast multiple plausible future motions given a single pose sequence from the past. Most previous works focus on designing elaborate losses to improve the accuracy, while the diversity is typically characterized by randomly sampling a set of latent variables from the latent prior, which is then decoded into possible motions. This joint training of sampling and decoding, however, suffers from posterior collapse as the learned latent variables tend to be ignored by a strong decoder, leading to limited diversity. Alternatively, inspired by the diffusion process in nonequilibrium thermodynamics, we propose MotionDiff, a diffusion probabilistic model to treat the kinematics of human joints as heated particles, which will diffuse from original states to a noise distribution. This process not only offers a natural way to obtain the "whitened'' latents without any trainable parameters, but also introduces a new noise in each diffusion step, both of which facilitate more diverse motions. Human motion prediction is then regarded as the reverse diffusion process that converts the noise distribution into realistic future motions conditioned on the observed sequence. Specifically, MotionDiff consists of two parts: a spatial-temporal transformer-based diffusion network to generate diverse yet plausible motions, and a flexible refinement network to further enable geometric losses and align with the ground truth. Experimental results on two datasets demonstrate that our model yields the competitive performance in terms of both diversity and accuracy.

Image embedding for denoising generative models

Denoising Diffusion models are gaining increasing popularity in the field of generative modeling for several reasons, including the simple and stable training, the excellent generative quality, and the solid probabilistic foundation. In this article, we address the problem of embedding an image into the latent space of Denoising Diffusion Models, that is finding a suitable “noisy” image whose denoising results in the original image. We particularly focus on Denoising Diffusion Implicit Models due to the deterministic nature of their reverse diffusion process. As a side result of our investigation, we gain a deeper insight into the structure of the latent space of diffusion models, opening interesting perspectives on its exploration, the definition of semantic trajectories, and the manipulation/conditioning of encodings for editing purposes. A particularly interesting property highlighted by our research, which is also characteristic of this class of generative models, is the independence of the latent representation from the networks implementing the reverse diffusion process. In other words, a common seed passed to different networks (each trained on the same dataset), eventually results in identical images.