Synthesis Tasks Research Articles

DISA: Disentangled Dual-Branch Framework for Affordance-Aware Human Insertion

Affordance-aware human insertion is a controllable human synthesis task aimed at seamlessly integrating a person into a scene while aligning human pose with contextual scene affordance and preserving human visual identity. Previous methods, typically reliant on a general framework of inpainting that injects all conditional information into a single branch, often struggle with the complexities of real-world contexts and the nuanced attributes of human figures. To this end, we present a novel DIS entangled dual-branch framework for A ffordance-aware human insertion task, termed as DISA, which focuses on both scene context comprehension and precise person attribute extraction. Specifically, our dual-branch design facilitates diffusion models to ensure disentangled and precise manipulations: one branch utilizes an additional network for deep scene context comprehension and control, while the other branch employs a parallel encoder to extract the feature of the reference person and injects this information through cross-attention mechanism. Furthermore, to comprehensively evaluate affordance-aware human insertion task, we introduce a new metric to assess the preservation of visual identity. We conduct a broad variety of evaluation experiments and validate the diversity and robustness of our method in different settings and downstream applications. Both qualitative and quantitative experimental analysis demonstrates that our approach outperforms previous methods in terms of image quality, pose accuracy and visual identity preservation.

Application of Transformers to Chemical Synthesis

Efficient chemical synthesis is critical for the production of organic chemicals, particularly in the pharmaceutical industry. Leveraging machine learning to predict chemical synthesis and improve the development efficiency has become a significant research focus in modern chemistry. Among various machine learning models, the Transformer, a leading model in natural language processing, has revolutionized numerous fields due to its powerful feature-extraction and representation-learning capabilities. Recent applications demonstrated that Transformer models can also significantly enhance the performance in chemical synthesis tasks, particularly in reaction prediction and retrosynthetic planning. This article provides a comprehensive review of the applications and innovations of Transformer models in the qualitative prediction tasks of chemical synthesis, with a focus on technical approaches, performance advantages, and the challenges associated with applying the Transformer architecture to chemical reactions. Furthermore, we discuss the future directions for improving the applications of Transformer models in chemical synthesis.

Data Augmentation for Deep Learning-Based Speech Reconstruction Using FOC-Based Methods

Neural audio reconstruction is an important subtopic of Neural Audio Synthesis (NAS), which is a current emerging topic of modern Artificial Intelligence (AI) applications. The objective of a neural audio reconstruction model is to achieve a viable audio waveform from an audio feature representation that excludes the phase information. Since the data-dependent nature of such systems demands an increased quantity of data, methods of increasing the quantity of data for neural network training arise as a topic of substantial interest. Although the applications of data augmentation methods for classification tasks are well documented, there is still room for development for applications of such methods on signal synthesis tasks. Additionally, the Fractional-Order Calculus (FOC) framework provides possibilities for quality applications for the signal processing domain. Still, it is important to show that the methods based on the FOC framework can be applied to different application domains to show the capabilities of this framework. In this paper, FOC-based methods are applied to a speech dataset for data augmentation purposes to increase the audio reconstruction performance of a neural network, a spectral consistency-based neural audio reconstruction model called Deep Griffin-Lim Iteration (DeGLI), with respect to objective measures PESQ and STOI. An FOC-based method for rescaling linear frequency for augmenting magnitude spectrogram data is proposed. Furthermore, together with an FOC-based phase estimation method, it is shown that an augmentation strategy that has the objective of increased spectral consistency should be considered in data augmentation for audio reconstruction tasks. The test results reveal that this type of strategy increases the performance of a spectral consistency-based neural audio reconstruction model by over 13% for smaller depths.

Coupling of state space modules and attention mechanisms: An input-aware multi-contrast MRI synthesis method.

Medical imaging plays a pivotal role in the real-time monitoring of patients during the diagnostic and therapeutic processes. However, in clinical scenarios, the acquisition of multi-modal imaging protocols is often impeded by a number of factors, including time and economic costs, the cooperation willingness of patients, imaging quality, and even safety concerns. We proposed a learning-based medical image synthesis method to simplify the acquisition of multi-contrast MRI. We redesigned the basic structure of the Mamba block and explored different integration patterns between Mamba layers and Transformer layers to make it more suitable for medical image synthesis tasks. Experiments were conducted on the IXI (a total of 575 samples, training set: 450 samples; validation set: 25 samples; test set: 100 samples) and BRATS (a total of 494 samples, training set: 350 samples; validation set: 44 samples; test set: 100 samples) datasets to assess the synthesis performance of our proposed method in comparison to some state-of-the-art models on the task of multi-contrast MRI synthesis. Our proposed model outperformed other state-of-the-art models in some multi-contrast MRI synthesis tasks. In the synthesis task from T1 to PD, our proposed method achieved the peak signal-to-noise ratio (PSNR) of 33.70 dB (95% CI, 33.61, 33.79) and the structural similarity index (SSIM) of 0.966 (95% CI, 0.964, 0.968). In the synthesis task from T2 to PD, the model achieved a PSNR of 33.90 dB (95% CI, 33.82, 33.98) and SSMI of 0.971 (95% CI, 0.969, 0.973). In the synthesis task from FLAIR to T2, the model achieved PSNR of 30.43 dB (95% CI, 30.29, 30.57) and SSIM of 0.938 (95% CI, 0.935, 0.941). Our proposed method could effectively model not only the high-dimensional, nonlinear mapping relationships between the magnetic signals of the hydrogen nucleus in tissues and the proton density signals in tissues, but also of the recovery process of suppressed liquid signals in FLAIR. The model proposed in our work employed distinct mechanisms in the synthesis of images belonging to normal and lesion samples, which demonstrated that our model had a profound comprehension of the input data. We also proved that in a hierarchical network, only the deeper self-attention layers were responsible for directing more attention on lesion areas.

Artificial intelligence for geometry-based feature extraction, analysis and synthesis in artistic images: a survey

Artificial Intelligence significantly enhances the visual art industry by analyzing, identifying and generating digitized artistic images. This review highlights the substantial benefits of integrating geometric data into AI models, addressing challenges such as high inter-class variations, domain gaps, and the separation of style from content by incorporating geometric information. Models not only improve AI-generated graphics synthesis quality, but also effectively distinguish between style and content, utilizing inherent model biases and shared data traits. We explore methods like geometric data extraction from artistic images, the impact on human perception, and its use in discriminative tasks. The review also discusses the potential for improving data quality through innovative annotation techniques and the use of geometric data to enhance model adaptability and output refinement. Overall, incorporating geometric guidance boosts model performance in classification and synthesis tasks, providing crucial insights for future AI applications in the visual arts domain.

Bilinear Models of Parts and Appearances in Generative Adversarial Networks.

Recent advances in the understanding of Generative Adversarial Networks (GANs) have led to remarkable progress in visual editing and synthesis tasks, capitalizing on the rich semantics that are embedded in the latent spaces of pre-trained GANs. However, existing methods are often tailored to specific GAN architectures and are limited to either discovering global semantic directions that do not facilitate localized control, or require some form of supervision through manually provided regions or segmentation masks. In this light, we present an architecture-agnostic approach that jointly discovers factors representing spatial parts and their appearances in an entirely unsupervised fashion. These factors are obtained by applying a semi-nonnegative tensor factorization on the feature maps, which in turn enables context-aware local image editing with pixel-level control. In addition, we show that the discovered appearance factors correspond to saliency maps that localize concepts of interest, without using any labels. Experiments on a wide range of GAN architectures and datasets show that, in comparison to the state of the art, our method is far more efficient in terms of training time and, most importantly, provides much more accurate localized control.

Measurement Guidance in Diffusion Models: Insight from Medical Image Synthesis.

In the field of healthcare, the acquisition of sample is usually restricted by multiple considerations, including cost, labor- intensive annotation, privacy concerns, and radiation hazards, therefore, synthesizing images-of-interest is an important tool to data augmentation. Diffusion models have recently attained state-of-the-art results in various synthesis tasks, and embedding energy functions has been proved that can effectively guide the pre-trained model to synthesize target samples. However, we notice that current method development and validation are still limited to improving indicators, such as Fréchet Inception Distance score (FID) and Inception Score (IS), and have not provided deeper investigations on downstream tasks, like disease grading and diagnosis. Moreover, existing classifier guidance which can be regarded as a special case of energy function can only has a singular effect on altering the distribution of the synthetic dataset. This may contribute to in-distribution synthetic sample that has limited help to downstream model optimization. All these limitations remind that we still have a long way to go to achieve controllable generation. In this work, we first conducted an analysis on previous guidance as well as its contributions on further applications from the perspective of data distribution. To synthesize samples which can help downstream applications, we then introduce uncertainty guidance in each sampling step and design an uncertainty-guided diffusion models. Extensive experiments on four medical datasets, with ten classic networks trained on the augmented sample sets provided a comprehensive evaluation on the practical contributions of our methodology. Furthermore, we provide a theoretical guarantee for general gradient guidance in diffusion models, which would benefit future research on investigating other forms of measurement guidance for specific generative tasks.

Plasma CycleGAN: Integrating Blood‐based Biomarkers for Cross‐modality Translation from MRI to PET

AbstractBackgroundCross‐modality translation between MRI and PET presents significant challenges due to the distinct mechanisms underlying those modalities. Currently, the leading method for this translation is CycleGAN. Previous studies have proved a strong correlation between BBBMs and brain amyloid status measured by PET. However, the impact of blood‐based biomarkers (BBBMs) on PET image synthesis has not yet been thoroughly evaluated. In this study, we propose Plasma CycleGAN, a generative model based on CycleGAN to synthesize PET images from MRI, while also incorporating BBBMs and other clinical covariates. We show that Plasma CycleGAN outperforms the state‐of‐the‐art CycleGAN for MRI‐to‐PET translation. Our method is the first to integrate BBBMs in a conditional cross modality translation from MRI to PET.MethodOur model was built on CycleGAN architecture, while the backbone was improved by integrating BBBMs as additional conditions in the input domain. To be specific, we introduce a masked perturbation in the MRI image, whose voxel‐level magnitudes are learnable parameters in the neural network. The dataset we used to evaluate our model is downloaded from the ADNI database. The cohort contains 160 subjects with matching BBBM, T1‐weighted MRI images, amyloid PET, and clinical features, including age, sex, weights, etc. The synthesized PET images are compared with the ground truth PET images for performance evaluation using traditional metrics like structural similarity index (SSIM), peak‐signal‐to‐noise ratio (PSNR) and mean square error (MSE). In addition, the Pearson correlation coefficients (PCCs) of the masked SUVR ratios were calculated.ResultWe evaluated the baseline CycleGAN model using MRI input, as well as combined inputs of MRI+BBBM and other clinical features. Our model achieved an SSIM of 0.72±0.16 and PSNR of 22.8±2.3 in PET image synthesis task using MRI+ Aβ42/40, both yielding the best results among all input combinations. MRI+sex achieved the best result using MSE measurement. PCCs of all experimental results ranged from 0.44 to 0.5.ConclusionWe introduce Plasma CycleGAN, the first generative model to integrate BBBMs and clinical features in PET image synthesis from MRI images. In PET image synthesis task, our proposed integrated model outperforms the baseline CycleGAN model.

Optimization-based process synthesis by phenomena-based building blocks and an MINLP-framework featuring structural screening

An existing approach for optimization-based process synthesis with abstracted phenomena-based building blocks (PBB) is extended by implementing it into a novel MINLP framework with structural screening. Consistency across the multilayer MINLP framework is guaranteed by creating a MathML/XML data model and subsequently exporting the code to the different program parts. The novel framework focuses both on fidelity by implementing thermodynamically sound models and on generality by employing a state-space superstructure that spans a large search space. In order to retain tractability, we insert a structural screening layer which pre-screens based on binary decision variables of the superstructure by graph- and rule-based analyses, penalizing non-physical instances without solution of the underlying MINLP. The MINLP framework is successfully applied on two challenging synthesis tasks to determine the separation of the feed streams of benzene and toluene, as well as of n-pentane, n-hexane, and n-heptane utilizing superstructures with two, respectively four PBB.

Neural Solving Uninterpreted Predicates with Abstract Gradient Descent

Uninterpreted predicate solving is a fundamental problem in formal verification, including loop invariant and constrained horn clauses predicate solving. Existing approaches have been mostly in symbolic ways. While achieving sustainable progress, they still suffer from inefficiency and seem unable to leverage the ever-increasing computility, such as GPU. Recently, neural relaxation has been proposed to tackle this problem. They treat the uninterpreted predicate-solving task as an optimization problem by relaxing the discrete search process into a learning process of neural networks. However, two bottlenecks keep them from being valid. First, relaxed neural networks cannot match the original semantics of predicates rigorously; second, the neural networks are difficult to train to reach global optimization. Therefore, this article presents a novel discrete neural architecture with the Abstract Gradient Decent (AGD) algorithm to directly solve uninterpreted predicates in the discrete hypothesis space. The abstract gradient is for discrete neurons whose calculation rules are designed in an abstract domain. Our approach conforms to the original semantics of predicates, and the proposed AGD algorithm can achieve global optimization satisfactorily. We implement the tool Dasp in the Boxes abstract domain to solve uninterpreted predicates in the QF-NIA SMT theory. In the experiments, Dasp has outperformed seven state-of-the-art tools across three predicate synthesis tasks.

Multisensory objects’ role on creativity

In this research, we investigated the role of multisensorial manipulation on creativity, and the influence of inspirational objects on creative outcomes. Object manipulation may support embodied cognition during a generative creative phase (emergence of motor, spatial, emotional ideas, etc.) then exploratory phase (creative fixation, development of a functional creation, etc.). Our protocol involved 136 engineering students divided into 34 groups which were provided with inspirational cubes illustrating manufacturing inventive principles or basic volumes from the Creative Mental Synthesis Task. They could manipulate these objects either in a visuo-haptic condition, or in a visuo-imaginative condition. Our results highlighted a main effect of manipulation, showing that visual-haptic condition led to higher creativity than visual-imaginative condition. We also observed several effects in favor of inspirational cubes with regard to basic volumes: significantly higher creativity, more subjective and inter-subjective facilitation behaviors, more cognitive and emotional operations. Participants also showed at an individual level a better mobilization of the multisensorial senses. Creative thinking may be stimulated when an active manipulation phase is set up before the creative production. This could contribute to improving practice for engineers, particularly for using additive manufacturing and/or during their training at school.

A Review of Facial Generation Technology Research

Abstract. Face generation, as a cutting-edge generative technology, has made significant strides in various image synthesis tasks, including facial inpainting, text-to-image translation, and video-based facial animation. Generating realistic and diverse human faces is a critical task in computer vision, with a wide array of real-world applications. Given the remarkable success of face generation models, there has been a growing interest in leveraging advanced techniques to further improve the quality, diversity, and control of generated faces. This paper will provide a comprehensive overview of the state-of-the-art face generation techniques. Specifically, the paper reviews the key approaches in this field, including Generative Adversarial Networks (GANs), Vector Quantized methods, and the more recent Diffusion Models, each of which has contributed significantly to the advancement of face generation. The paper discusses how these models have evolved to handle the complexity of face generation, from capturing subtle facial details to enabling fine-grained control over facial attributes. The paper also explores the major applications of face generation technologies, with a particular emphasis on their use in entertainment, virtual reality, and security. Finally, the paper identifies promising directions for future research in face generation, such as improving the interpretability of models, addressing ethical concerns, and enhancing the ability to generate faces that are both highly realistic and diverse.

VPRF: Visual Perceptual Radiance Fields for Foveated Image Synthesis.

Neural radiance fields (NeRF) has achieved revolutionary breakthrough in the novel view synthesis task for complex 3D scenes. However, this new paradigm struggles to meet the requirements for real-time rendering and high perceptual quality in virtual reality. In this paper, we propose VPRF, a novel visual perceptual based radiance fields representation method, which for the first time integrates the visual acuity and contrast sensitivity models of human visual system (HVS) into the radiance field rendering framework. Initially, we encode both the appearance and visual sensitivity information of the scene into our radiance field representation. Then, we propose a visual perceptual sampling strategy, allocating computational resources according to the HVS sensitivity of different regions. Finally, we propose a sampling weight-constrained training scheme to ensure the effectiveness of our sampling strategy and improve the representation of the radiance field based on the scene content. Experimental results demonstrate that our method renders more efficiently, with higher PSNR and SSIM in the foveal and salient regions compared to the state-of-the-art FoV-NeRF. The results of the user study confirm that our rendering results exhibit high-fidelity visual perception.

Spatial and Modal Optimal Transport for Fast Cross-Modal MRI Reconstruction.

Multi-modal magnetic resonance imaging (MRI) plays a crucial role in comprehensive disease diagnosis in clinical medicine. However, acquiring certain modalities, such as T2-weighted images (T2WIs), is time-consuming and prone to be with motion artifacts. It negatively impacts subsequent multi-modal image analysis. To address this issue, we propose an end-to-end deep learning framework that utilizes T1-weighted images (T1WIs) as auxiliary modalities to expedite T2WIs' acquisitions. While image pre-processing is capable of mitigating misalignment, improper parameter selection leads to adverse pre-processing effects, requiring iterative experimentation and adjustment. To overcome this shortage, we employ Optimal Transport (OT) to synthesize T2WIs by aligning T1WIs and performing cross-modal synthesis, effectively mitigating spatial misalignment effects. Furthermore, we adopt an alternating iteration framework between the reconstruction task and the cross-modal synthesis task to optimize the final results. Then, we prove that the reconstructed T2WIs and the synthetic T2WIs become closer on the T2 image manifold with iterations increasing, and further illustrate that the improved reconstruction result enhances the synthesis process, whereas the enhanced synthesis result improves the reconstruction process. Finally, experimental results from FastMRI and internal datasets confirm the effectiveness of our method, demonstrating significant improvements in image reconstruction quality even at low sampling rates.

WaterHE-NeRF: Water-ray matching neural radiance fields for underwater scene reconstruction

Neural Radiance Field (NeRF) technology demonstrates immense potential in novel viewpoint synthesis tasks due to its physics-based volumetric rendering process, which is particularly promising in underwater scenes. However, existing underwater NeRF methods face challenges in handling light attenuation caused by the water medium and the lack of real Ground Truth (GT) supervision. To address these issues, we propose WaterHE-NeRF, a novel approach incorporating a water-ray matching field developed based on Retinex theory. This field precisely encodes color, density, and illuminance attenuation in three-dimensional space. WaterHE-NeRF employs an illuminance attenuation mechanism to generate degraded and clear multi-view images, optimizing image restoration by combining reconstruction loss with Wasserstein distance. Furthermore, using histogram equalization (HE) as pseudo-GT, WaterHE-NeRF enhances the network’s accuracy in preserving original details and color distribution. Extensive experiments on real underwater and synthetic datasets validate the effectiveness of WaterHE-NeRF.

Using Machine Learning for Systematic Literature Review Case in Point: Agile Software Development

ABSTRACTSystematic literature reviews (SLRs) are essential for researchers to keep up with past and recent research in their domains. However, the rapid growth in knowledge creation and the rising number of publications have made this task increasingly complex and challenging. Moreover, most systematic literature reviews are performed manually, which requires significant effort and creates potential bias. The risk of bias is particularly relevant in the data synthesis task, where researchers interpret each study's evidence and summarize the results. This study uses an experimental approach to explore using machine learning (ML) techniques in the SLR process. Specifically, this study replicates a study that manually performed sentiment analysis for the data synthesis step to determine the polarity (negative or positive) of evidence extracted from studies in the field of agile methodology. This study employs a lexicon‐based approach to sentiment analysis and achieves an accuracy rate of approximately 86.5% in identifying study evidence polarity.

GLTScene: Global‐to‐Local Transformers for Indoor Scene Synthesis with General Room Boundaries

AbstractWe present GLTScene, a novel data‐driven method for high‐quality furniture layout synthesis with general room boundaries as conditions. This task is challenging since the existing indoor scene datasets do not cover the variety of general room boundaries. We incorporate the interior design principles with learning techniques and adopt a global‐to‐local strategy for this task. Globally, we learn the placement of furniture objects from the datasets without considering their alignment. Locally, we learn the alignment of furniture objects relative to their nearest walls, according to the alignment principle in interior design. The global placement and local alignment of furniture objects are achieved by two transformers respectively. We compare our method with several baselines in the task of furniture layout synthesis with general room boundaries as conditions. Our method outperforms these baselines both quantitatively and qualitatively. We also demonstrate that our method can achieve other conditional layout synthesis tasks, including object‐level conditional generation and attribute‐level conditional generation. The code is publicly available at https://github.com/WWalter-Lee/GLTScene.

Integrating hybrid priors for face photo-sketch synthesis

Benefiting from the advancement of deep learning techniques, face photo-sketch synthesis has witnessed significant progress in recent years. Cutting-edge methods typically treat this task as an image-to-image translation problem and train a conditional generative model to learn the mapping between two domains. However, purely parametric deep learning models often struggle to capture instance-level details due to limited training samples and tend to focus on domain-level mapping. Moreover, sketch-to-photo synthesis is more challenging than photo-to-sketch synthesis and holds greater significance in the realm of public security, but it has not been well-studied in existing methods. To address these challenges, we introduce an innovative framework that synergistically integrates parametric and non-parametric approaches, infusing facial generative priors and instance-level prior knowledge from the target domain to enrich texture detail synthesis. Specifically, our framework employs a semantic-aware network to facilitate coarse cross-domain reconstruction, thereby capturing domain-level information. Moreover, through efficient neural patch matching between the input image and multiple reference (training) samples, we can harness instance-level prior knowledge as a detailed texture representation to enhance detail fidelity. For the sketch-to-photo synthesis task, we further propose a local patch correspondence mechanism that improves the rationality of matching through local constraint. To further enhance the generation of realistic and detailed facial features, we incorporate a pre-trained StyleGAN as the decoder, leveraging its extensive facial generative priors. Additionally, we introduce the relaxed Earth Movers Distance (rEMD) loss to improve the style consistency between the generated results and the target domain. Extensive experiments show that our method achieves state-of-the-art performance on both quantitative and qualitative evaluations.

Music-stylized hierarchical dance synthesis with user control

BackgroundSynthesizing dance motions to match musical inputs is a significant challenge in animation research. Compared to functional human motions, such as locomotion, dance motions are creative and artistic, often influenced by music, and can be independent body language expressions. Dance choreography requires motion content to follow a general dance genre, whereas dance performances under musical influence are infused with diverse impromptu motion styles. Considering the high expressiveness and variations in space and time, providing accessible and effective user control for tuning dance motion styles remains an open problem. MethodsIn this study, we present a hierarchical framework that decouples the dance synthesis task into independent modules. We use a high-level choreography module built as a Transformer-based sequence model to predict the long-term structure of a dance genre and a low-level realization module that implements dance stylization and synchronization to match the musical input or user preferences. This novel framework allows the individual modules to be trained separately. Because of the decoupling, dance composition can fully utilize existing high-quality dance datasets that do not have musical accompaniments, and the dance implementation can conveniently incorporate user controls and edit motions through a decoder network. Each module is replaceable at runtime, which adds flexibility to the synthesis of dance sequences. ResultsSynthesized results demonstrate that our framework generates high-quality diverse dance motions that are well adapted to varying musical conditions and user controls.

Квазиоптимальный закон управления для лагранжевых систем с дефицитом управляющих воздействий

Mechanical systems with a deficit of control influences are essentially nonlinear dynamic systems of high order, which are characterized by the presence of significant dynamic interaction between the elements. Therefore, the task of control synthesis should be considered in the initial nonlinear formulation, since the use of linear synthesis methods can lead to a loss of stability or significantly worsen the quality of the control process. When solving the problem of synthesis with a deficit of control influences, it is advisable to analyze such a mechanical system as a non-stationary one and use the principle of decomposition in conjunction with the maximum condition of generalized power, which makes it possible to find the structure of a universal two-level control system. The resulting quasi-optimal control provides an increase in the stability area and the quality of the transient process in terms of a quadratic criterion in comparison with known linear algorithms.