Realistic Synthesis Research Articles

Glioblastoma Synthesis and Segmentation with 3D Multi-Modal MRI: A Study using Generative Adversarial Networks

The Grade IV cancer Glioblastoma is an extremely common and aggressive brain tumour. It is of significant consequence that histopathologic examinations should be able to identify and capture the tumour’s genetic variability for assistance in treatment. The use of Deep Learning - in particular CNNs and GANs - have become prominent in dealing with various image segmentation and detection tasks. The use of GANs have another importance - to expand the available training set by generating realistic pseudomedical images. Multi-modal MRIs, moreover, are also crucial as they lead to more successful performances. Nonetheless, accurate segmentation and realistic image synthesis remain challenging tasks. In this study, the history and various breakthroughs/challenges of utilising deep learning in glioblastoma detection is outlined and evaluated. To see networks in action, an adjusted and calibrated Vox2Vox network - a 3D implementation of the Pix2Pix translator - is trained on the biggest public brain tumour dataset BraTS 2020. The experimental results demonstrate the versatility and improvability of GAN networks in both fields of augmentation and segmentation. Overall, deep learning in medical imaging remains an extremely intoxicating field full of meticulous and innovative new studies.

Erratum to "Realistic Lung Nodule Synthesis With Multi-Target Co-Guided Adversarial Mechanism".

In the above article [1], Figs. 5 and 6 were published incorrectly. The correct figures are given below.

Bus journey simulation to develop public transport predictive algorithms

Encouraging the use of public transport is essential to combat congestion and pollution in an urban environment. To achieve this, the reliability of arrival time prediction should be improved as this is one area of improvement frequently requested by passengers. The development of accurate predictive algorithms requires good quality data, which is often not available. Here we demonstrate a method to synthesise data using a reference curve approach derived from very limited real world data without reliable ground truth. This approach allows the controlled introduction of artefacts and noise to simulate their impact on prediction accuracy. To illustrate these impacts, a recurrent neural network next-step prediction is used to compare different scenarios in two different UK cities. The results show that a realistic data synthesis is possible, allowing for controlled testing of predictive algorithms. It also highlights the importance of reliable data transmission to gain such data from real world sources. Our main contribution is the demonstration of a synthetic data generator for public transport data, which can be used to compensate for low data quality. We further show that this data generator can be used to develop and enhance predictive algorithms in the context of urban bus networks if high-quality data is limited, by mixing synthetic and real data. • Method to generate synthetic bus journeys based on very limited and unreliable data. • Insights into data quality challenges in public transport systems. • Algorithms can be improved by training on a mixture of synthetic and real-world data. • The demonstrated data synthesis could be the base for a public transport benchmarking dataset.

Synthetic region screening and adaptive feature fusion for constructing a flexible object detection database

A large amount of training data (including samples and labels) are required to achieve the task of object detection. However, the construction of databases generally has several problems. The samples are not balanced. In some scenarios, very few or even no samples can be collected; manual collection and labeling are expensive. Obtaining samples from public object detection data sets is simple, but the categories that can be obtained are limited and cannot meet the needs of other detection tasks. Therefore, we propose a method to automatically construct a flexible object detection database and complete automatic labeling. Our main advantage lies in the proposed synthetic region screening and adaptive feature fusion algorithm, which generates sample images and corresponding annotation files that are similar to the real collection. Our algorithm has a stronger and more realistic synthesis ability for three-dimensional objects than other data synthesis methods. Our method improves the system’s object detection accuracy in the following ways: (1) it synthesizes a wider range of images for certain categories with few samples in the data set to solve the problem of sample imbalance; (2) it adds new categories to the public data set to meet the needs of rapid deployment of special object detection tasks; and (3) it fuses different foregrounds and backgrounds to enrich the diversity of database samples. To effectively evaluate the proposed method, we conducted experiments separately, and the experimental results proved that our method is superior to other existing data synthesis and data enhancement methods. Combining the generated samples with the real data set, the object detection accuracy value increased from 9.47% to 58.92%. Extending the objects of the self-collected data set to the public data set and automatically generating high-quality annotations, the detection accuracy of extended objects reached more than 50%.

Image synthesis via adversarial geometric consistency pursuit

Realistic image synthesis is a class of vision problems where the goal is to synthesize new images by fusing a source image and a target image. It is a challenging problem due to the visual gap between the two images, including geometry and appearance. To synthetically address the geometric distortion and appearance realism, we propose a novel method, the adversarial geometric consistency pursuit model (AGCP), which explicitly allows seamless image synthesis. The proposed method takes geometric correction and appearance harmonization into account, handling the relative scaling, spatial layout, color, viewpoint, and distortion transformation to generate a realistic composite image. Moreover, we also propose a joint geometric consistency pursuit loss that handles the geometric consistency and enhances the network to generalize better for different scales of source images. Our comparative evaluation demonstrates the effectiveness of the proposed method in the aforementioned challenging cases.

Realistic Lung Nodule Synthesis With Multi-Target Co-Guided Adversarial Mechanism.

The important cues for a realistic lung nodule synthesis include the diversity in shape and background, controllability of semantic feature levels, and overall CT image quality. To incorporate these cues as the multiple learning targets, we introduce the Multi-Target Co-Guided Adversarial Mechanism, which utilizes the foreground and background mask to guide nodule shape and lung tissues, takes advantage of the CT lung and mediastinal window as the guidance of spiculation and texture control, respectively. Further, we propose a Multi-Target Co-Guided Synthesizing Network with a joint loss function to realize the co-guidance of image generation and semantic feature learning. The proposed network contains a Mask-Guided Generative Adversarial Sub-Network (MGGAN) and a Window-Guided Semantic Learning Sub-Network (WGSLN). The MGGAN generates the initial synthesis using the mask combined with the foreground and background masks, guiding the generation of nodule shape and background tissues. Meanwhile, the WGSLN controls the semantic features and refines the synthesis quality by transforming the initial synthesis into the CT lung and mediastinal window, and performing the spiculation and texture learning simultaneously. We validated our method using the quantitative analysis of authenticity under the Fréchet Inception Score, and the results show its state-of-the-art performance. We also evaluated our method as a data augmentation method to predict malignancy level on the LIDC-IDRI database, and the results show that the accuracy of VGG-16 is improved by 5.6%. The experimental results confirm the effectiveness of the proposed method.

A METHOD FOR SYNTHESIZING THERMAL IMAGES USING GAN MULTI-LAYERED APPROACH

Abstract. The active development of neural network technologies and optoelectronic systems has led to the introduction of computer vision technologies in various fields of science and technology. Deep learning made it possible to solve complex problems that a person had not been able to solve before. The use of multi-spectral optical systems has significantly expanded the field of application of video systems. Tasks such as image recognition, object re-identification, video surveillance require high accuracy, speed and reliability. These qualities are provided by algorithms based on deep convolutional neural networks. However, they require to have large databases of multi-spectral images of various objects to achieve state-of-the-art results. While large and various databases of color images of different objects are widely available in public domain, then similar databases of thermal images are either not available, or they represent a small number of types of objects. The quality of three-dimensional modeling for the thermal imaging spectral range remains at an insufficient level for solving a number of important tasks, which require high precision and reliability. The realistic synthesis of thermal images is especially important due to the complexity and high cost of obtaining real data. This paper is focused on the development of a method for synthesizing thermal imaging images based on generative adversarial neural networks. We developed an algorithm for a multi-spectral image-to-image translation. We have changed to the original GAN architecture and converted the loss function. We presented a new learning approach. For this, we prepared a special training dataset including about 2000 image tensors. The evaluation of the results obtained showed that the proposed method can be used to expand the available databases of thermal images.

Example-Based Facial Animation of Virtual Reality Avatars Using Auto-Regressive Neural Networks.

This article presents a hybrid animation approach that combines example-based and neural animation methods to create a simple, yet powerful animation regime for human faces. Example-based methods usually employ a database of prerecorded sequences that are concatenated or looped in order to synthesize novel animations. In contrast to this traditional example-based approach, we introduce a light-weight auto-regressive network to transform our animation-database into a parametric model. During training, our network learns the dynamics of facial expressions, which enables the replay of annotated sequences from our animation database as well as their seamless concatenation in new order. This representation is especially useful for the synthesis of visual speech, where coarticulation creates interdependencies between adjacent visemes, which affects their appearance. Instead of creating an exhaustive database that contains all viseme variants, we use our animation-network to predict the correct appearance. This allows realistic synthesis of novel facial animation sequences like visual-speech but also general facial expressions in an example-based manner.

Design of Metals Sulfides with Carbon Materials for Supercapacitor Applications: A Review

Supercapacitors are very exclusive among energy‐storage devices for the wearable and portable industry. Due to simplicity, abundance occurrence, good electrical conductivity, and rich redox states, metal sulfides have a significant impact in supercapacitor applications in the present decade. Sulfur is the anion, associated with metal cations, and provides substantial tunable electrochemical, electrical, physical, and chemical properties in metal sulfides. In addition, carbonaceous materials enhance the electrochemical characteristics of metal sulfides synergistically for supercapacitors. Fast and scalable synthesis and fabrication of the supercapacitor cell with low‐cost metal sulfides and heterostructures should be developed to achieve the potential applications of these hybrid glorious materials. Herein, the significant and critical discussion of metal sulfides synthesis and their hybrid with carbon electrode structures for supercapacitor applications is disclosed. The realistic synthesis of metal sulfides, basic charge storage phenomena to elaborate the electrode nature, electrochemical properties including a single electrode, and asymmetric and symmetric cells are demonstrated for comparative purposes. The market values and application of supercapacitors are also elaborated. Herein, a better understanding to fresh readers about metal sulfide hybrid electrode structures for supercapacitors is provided.

Generative adversarial learning for detail-preserving face sketch synthesis

Face sketch synthesis aims to generate a face sketch image from a corresponding photo image and has wide applications in law enforcement and digital entertainment. Despite the remarkable achievements that have been made in face sketch synthesis, most existing works pay main attention to the facial content transfer, at the expense of facial detail information. In this paper, we present a new generative adversarial learning framework to focus on detail preservation for realistic face sketch synthesis. Specifically, the high-resolution network is modified as generator to transform a face image from photograph to sketch domain. Except for the common adversarial loss, we design a detail loss to force the synthesized face sketch images have proximate details to its corresponding photo images. In addition, the style loss is adopted to restrain the synthesized face sketch images have vivid sketch style as the hand-drawn sketch images. Experimental results demonstrate that the proposed approach achieves superior performance, compared to state-of-the-art approaches, both on visual perception and objective evaluation. Specifically, this study indicated the higher FSIM values (0.7345 and 0.7080) and Scoot values (0.5317 and 0.5091) than most comparison methods on the CUFS and CUFSF datasets, respectively.

Transformation guided representation GAN for pose invariant face recognition

Face recognition is an important topic in the field of computer vision and has been a vital biometric technique for identity authentication. It is widely used in areas such as public security, military, and daily life. However, face recognition is inherently a challenging problem due to variations in poses, facial expressions, age, and occlusion. In this work, we propose a generative adversarial network (GAN) architecture that disentangles identity and pose variations to learn generative and discriminative representations for pose-invariant face recognition. We use an iterative warping scheme that achieves better results than with the use of a single generator. The features from the encoder are considered pose-invariant features for face recognition, and evaluations on databases demonstrate the usefulness of this approach over prior methods. For example, we report 97.0% (+ 12.7%) and 90.5% (+ 8.4%) accuracy on the Feret and Caspeal datasets compared to 78.2% achieved by the best method without warping. In particular, there are two notable novelties. First, the disentangled architecture GAN (D-GAN) performs frontal face synthesis via an encoder-decoder structure in the generator with the pose variations provided to the decoder and discriminator. Second, we utilize the generator encoder as a spatial transformer network that seeks realistic image synthesis in the geometric warp parameter space.

The Effect of Latent Space Dimension on the Quality of Synthesized Human Face Images

In recent years Generative Adversarial Networks (GANs) have achieved remarkable results in the task of realistic image synthesis. Despite their continued success and advances, there still lacks a thorough understanding of how precisely GANs map random latent vectors to realistic-looking images and how the priors set on the latent space affect the learned mapping. In this work, we analyze the effect of the chosen latent dimension on the final quality of synthesized images of human faces and learned data representations. We show that GANs can generate images plausibly even with latent dimensions significantly smaller than the standard dimensions like 100 or 512. Although one might expect that larger latent dimensions encourage the generation of more diverse and enhanced quality images, we show that an increase of latent dimension after some point does not lead to visible improvements in perceptual image quality nor in quantitative estimates of its generalization abilities.

Image Comes Dancing With Collaborative Parsing-Flow Video Synthesis.

Transferring human motion from a source to a target person poses great potential in computer vision and graphics applications. A crucial step is to manipulate sequential future motion while retaining the appearance characteristic. Previous work has either relied on crafted 3D human models or trained a separate model specifically for each target person, which is not scalable in practice. This work studies a more general setting, in which we aim to learn a single model to parsimoniously transfer motion from a source video to any target person given only one image of the person, named as Collaborative Parsing-Flow Network (CPF-Net). The paucity of information regarding the target person makes the task particularly challenging to faithfully preserve the appearance in varying designated poses. To address this issue, CPF-Net integrates the structured human parsing and appearance flow to guide the realistic foreground synthesis which is merged into the background by a spatio-temporal fusion module. In particular, CPF-Net decouples the problem into stages of human parsing sequence generation, foreground sequence generation and final video generation. The human parsing generation stage captures both the pose and the body structure of the target. The appearance flow is beneficial to keep details in synthesized frames. The integration of human parsing and appearance flow effectively guides the generation of video frames with realistic appearance. Finally, the dedicated designed fusion network ensure the temporal coherence. We further collect a large set of human dancing videos to push forward this research field. Both quantitative and qualitative results show our method substantially improves over previous approaches and is able to generate appealing and photo-realistic target videos given any input person image. All source code and dataset will be released at https://github.com/xiezhy6/CPF-Net.

Stabilization of \u03b5-iron carbide as high-temperature catalyst under realistic Fischer\u2013Tropsch synthesis conditions

The development of efficient catalysts for Fischer–Tropsch (FT) synthesis, a core reaction in the utilization of non-petroleum carbon resources to supply energy and chemicals, has attracted much recent attention. ε-Iron carbide (ε-Fe2C) was proposed as the most active iron phase for FT synthesis, but this phase is generally unstable under realistic FT reaction conditions (> 523 K). Here, we succeed in stabilizing pure-phase ε-Fe2C nanocrystals by confining them into graphene layers and obtain an iron-time yield of 1258 μmolCO gFe−1s−1 under realistic FT synthesis conditions, one order of magnitude higher than that of the conventional carbon-supported Fe catalyst. The ε-Fe2C@graphene catalyst is stable at least for 400 h under high-temperature conditions. Density functional theory (DFT) calculations reveal the feasible formation of ε-Fe2C by carburization of α-Fe precursor through interfacial interactions of ε-Fe2C@graphene. This work provides a promising strategy to design highly active and stable Fe-based FT catalysts.

Design and implementation of net zero displacement filter for the synthesis of a mechanical shock signal under specified shock response spectrum

Electronic and optical products are vulnerable under mechanical shock environment. Designed products need to be validated by shock testing and/or numerical simulation, using representative acceleration-time history signals. However, specifications derived from measurements are normally given in terms of shock response spectrum (SRS) without a corresponding time history signal, and therefore, there is a need to synthesis acceleration-time histories from a given SRS specification. This paper proposed a net zero displacement filter and a realistic time history synthesis method. By scaling a relevant field measurement, acceleration-time histories can be synthesized, which can meet the net zero displacement constrain and a given SRS specification within ±3 dB margin.

Augmenting Colonoscopy using Extended and Directional CycleGAN for Lossy Image Translation.

Colorectal cancer screening modalities, such as optical colonoscopy (OC) and virtual colonoscopy (VC), are critical for diagnosing and ultimately removing polyps (precursors of colon cancer). The non-invasive VC is normally used to inspect a 3D reconstructed colon (from CT scans) for polyps and if found, the OC procedure is performed to physically traverse the colon via endoscope and remove these polyps. In this paper, we present a deep learning framework, Extended and Directional CycleGAN, for lossy unpaired image-to-image translation between OC and VC to augment OC video sequences with scale-consistent depth information from VC, and augment VC with patient-specific textures, color and specular highlights from OC (e.g, for realistic polyp synthesis). Both OC and VC contain structural information, but it is obscured in OC by additional patient-specific texture and specular highlights, hence making the translation from OC to VC lossy. The existing CycleGAN approaches do not handle lossy transformations. To address this shortcoming, we introduce an extended cycle consistency loss, which compares the geometric structures from OC in the VC domain. This loss removes the need for the CycleGAN to embed OC information in the VC domain. To handle a stronger removal of the textures and lighting, a Directional Discriminator is introduced to differentiate the direction of translation (by creating paired information for the discriminator), as opposed to the standard CycleGAN which is direction-agnostic. Combining the extended cycle consistency loss and the Directional Discriminator, we show state-of-the-art results on scale-consistent depth inference for phantom, textured VC and for real polyp and normal colon video sequences. We also present results for realistic pendunculated and flat polyp synthesis from bumps introduced in 3D VC models.

Gender based face aging with cycle-consistent adversarial networks

Face aging is a task which referred to image synthesis, and the challenge comes from the training dataset, most existing face aging works require paired face images which is difficult to collect. Face images with various ages of the same person can be considered as unpaired images which come from different domains. The degree of aging effect can be influenced by age, gender, race and some other factors. In this paper, we are committed to studying the impact of gender on face aging problem, which involves the processing and modeling of face images.It has been proved that Generative Adversarial Networks(GANs) is competitive in realistic image synthesis, and many works employed Cycle-Consistent Adversarial Networks(CycleGANs) have shown the high performance in unpaired image-to-image translation.To overcome current difficulties and improve the performance of existing models on the face aging tasks, we proposed an innovative Gender-based training method using CycleGAN by pairwise training CycleGAN over several age groups which are grouped by age and gender. We build a constraint model based on gender discrimination to better simulate the expected aging effect of face images.To evaluate our works using subjective method, we have set up a quantitative evaluation mechanism with participants involved in. Compared with other similar subjective evaluation methods, our method is more objective in the demonstration of experimental results. The experimental results show that our method has a more realistic and excellent performance compared to those using CycleGAN for face age synthesis directly.

Adaptive 3D Model-Based Facial Expression Synthesis and Pose Frontalization.

Facial expressions are one of the important non-verbal ways used to understand human emotions during communication. Thus, acquiring and reproducing facial expressions is helpful in analyzing human emotional states. However, owing to complex and subtle facial muscle movements, facial expression modeling from images with face poses is difficult to achieve. To handle this issue, we present a method for acquiring facial expressions from a non-frontal single photograph using a 3D-aided approach. In addition, we propose a contour-fitting method that improves the modeling accuracy by automatically rearranging 3D contour landmarks corresponding to fixed 2D image landmarks. The acquired facial expression input can be parametrically manipulated to create various facial expressions through a blendshape or expression transfer based on the FACS (Facial Action Coding System). To achieve a realistic facial expression synthesis, we propose an exemplar-texture wrinkle synthesis method that extracts and synthesizes appropriate expression wrinkles according to the target expression. To do so, we constructed a wrinkle table of various facial expressions from 400 people. As one of the applications, we proved that the expression-pose synthesis method is suitable for expression-invariant face recognition through a quantitative evaluation, and showed the effectiveness based on a qualitative evaluation. We expect our system to be a benefit to various fields such as face recognition, HCI, and data augmentation for deep learning.

Highly efficient Co@NCS nanosheet electrocatalyst for oxygen reduction reaction: An environment-friendly, low-cost and sustainable electrocatalyst

Rational, low cost, stable and more realistic synthesis of electrocatalysts comprising of earth abundant metals alterative to platinum-based catalysts is highly required for fuel cells. In this work, we demonstrate 2D nanosheet catalyst from metal organic frame works with zinc and 2-amino thiazole as organic ligand in presence of cobalt metal ions. The special bonding configurations of zinc with 2-amino thiazole ligand resulted in formation of 2D ZnxCo1-x(C3H4N2S) and the pyrolysis of 2D ZnxCo1-x(C3H4N2S) resulted in Co@NSC electrocatalyst. The electrochemical oxygen reduction reaction analysis of the Co@NSC catalyst reveals excellent activity with the half-wave potential of 0.88 V, similar to the commercial Pt/C catalyst. Moreover, Co@NSC catalyst also show the ultra-low HO2− formation bellow < 0.5 %, along with negligible degradation after 5000 potential cycles. In a alkaline fuel cell conditions, the Co@NCS catalyst delivered a power density of 37 mW cm-2.

Human Synthesis and Scene Compositing

Generating good quality and geometrically plausible synthetic images of humans with the ability to control appearance, pose and shape parameters, has become increasingly important for a variety of tasks ranging from photo editing, fashion virtual try-on, to special effects and image compression. In this paper, we propose a HUSC (HUman Synthesis and Scene Compositing) framework for the realistic synthesis of humans with different appearance, in novel poses and scenes. Central to our formulation is 3d reasoning for both people and scenes, in order to produce realistic collages, by correctly modeling perspective effects and occlusion, by taking into account scene semantics and by adequately handling relative scales. Conceptually our framework consists of three components: (1) a human image synthesis model with controllable pose and appearance, based on a parametric representation, (2) a person insertion procedure that leverages the geometry and semantics of the 3d scene, and (3) an appearance compositing process to create a seamless blending between the colors of the scene and the generated human image, and avoid visual artifacts. The performance of our framework is supported by both qualitative and quantitative results, in particular state-of-the art synthesis scores for the DeepFashion dataset.