Conditional Adversarial Network Research Articles

Enhanced MRI-based brain tumour classification with a novel Pix2pix generative adversarial network augmentation framework.

The scarcity of medical imaging datasets and privacy concerns pose significant challenges in artificial intelligence-based disease prediction. This poses major concerns to patient confidentiality as there are now tools capable of extracting patient information by merely analysing patient's imaging data. To address this, we propose the use of synthetic data generated by generative adversarial networks as a solution. Our study pioneers the utilisation of a novel Pix2Pix generative adversarial network model, specifically the 'image-to-image translation with conditional adversarial networks,' to generate synthetic datasets for brain tumour classification. We focus on classifying four tumour types: glioma, meningioma, pituitary and healthy. We introduce a novel conditional deep convolutional neural network architecture, developed from convolutional neural network architectures, to process the pre-processed generated synthetic datasets and the original datasets obtained from the Kaggle repository. Our evaluation metrics demonstrate the conditional deep convolutional neural network model's high performance with synthetic images, achieving an accuracy of 86%. Comparative analysis with state-of-the-art models such as Residual Network50, Visual Geometry Group 16, Visual Geometry Group 19 and InceptionV3 highlights the superior performance of our conditional deep convolutional neural network model in brain tumour detection, diagnosis and classification. Our findings underscore the efficacy of our novel Pix2Pix generative adversarial network augmentation technique in creating synthetic datasets for accurate brain tumour classification, offering a promising avenue for improved disease prediction and treatment planning.

Multiscale Conditional Adversarial Networks based domain-adaptive method for rotating machinery fault diagnosis under variable working conditions

Deep learning has been increasingly used in health management and maintenance decision-making for rotating machinery. However, some challenges must be addressed to make this technology more effective. For example, the collected data is assumed to follow the same feature distribution, and sufficient labeled training data are available. Unfortunately, domain shifts occur inevitably in real-world scenarios due to different working conditions, and acquiring sufficient labeled samples is time-consuming and expensive in complex environments. This study proposes a novel domain adaptive framework called deep Multiscale Conditional Adversarial Networks (MCAN) for machinery fault diagnosis to address these shortcomings. The MCAN model comprises two key components. Constructed by a novel multiscale module with an attention mechanism, the first component is a shared feature generator that captures rich features at different internal perceptual scales, and the attention mechanism determines the weights assigned to each scale, enhancing the model’s dynamic adjustment and self-adaptation capabilities. The second component consists of two domain classifiers based on Bidirectional Long Short-Term Memory (BiLSTM) leveraging spatiotemporal features at various levels to achieve domain adaptation in the output space. The deep domain classifier also captures the cross-covariance dependencies between feature representations and classifier predictions, thereby improving the predictions’ discriminability. The proposed method has been evaluated using two publicly available fault diagnosis datasets and one condition monitoring experiment. The results of cross-domain transfer tasks demonstrated that the proposed method outperformed several state-of-the-art methods in terms of transferability and stability. This result is a significant step forward in deep learning for health management and maintenance decision-making for rotating machinery, and it has the potential to revolutionize its future application.

Domain Adaptation for Bearing Fault Diagnosis Based on SimAM and Adaptive Weighting Strategy.

Domain adaptation techniques are crucial for addressing the discrepancies between training and testing data distributions caused by varying operational conditions in practical bearing fault diagnosis. However, transfer fault diagnosis faces significant challenges under complex conditions with dispersed data and distinct distribution differences. Hence, this paper proposes CWT-SimAM-DAMS, a domain adaptation method for bearing fault diagnosis based on SimAM and an adaptive weighting strategy. The proposed scheme first uses Continuous Wavelet Transform (CWT) and Unsharp Masking (USM) for data preprocessing, and then feature extraction is performed using the Residual Network (ResNet) integrated with the SimAM module. This is combined with the proposed adaptive weighting strategy based on Joint Maximum Mean Discrepancy (JMMD) and Conditional Adversarial Domain Adaption Network (CDAN) domain adaptation algorithms, which minimizes the distribution differences between the source and target domains more effectively, thus enhancing domain adaptability. The proposed method is validated on two datasets, and experimental results show that it improves the accuracy of bearing fault diagnosis.

Deep conditional adversarial subdomain adaptation network for unsupervised mechanical fault diagnosis

In recent years, intelligent fault diagnosis technology has rapidly advanced, achieving remarkable results. Most of them assume that the source and target domains have similar distributions. However, the actual working conditions of rotating machinery are variable, leading to a significant distribution discrepancy between the source and target domains. The majority of existing domain adaptation methods mainly consider either the marginal distribution or the conditional distribution between the source and target domains, which limits the improvement of diagnostic performance in the target domain. Considering the discrepancy in domain distributions, a deep conditional adversarial subdomain adaptation network (DCASAN) is proposed in this paper for unsupervised mechanical fault diagnosis under unknown working conditions. In DCASAN, a backbone network based on the feature fusion of convolutional neural network (CNN) and graph convolutional network (GCN) is designed for feature extraction. Furthermore, a conditional adversarial subdomain adaptation strategy is proposed, which simultaneously considers both the marginal distribution and the conditional distribution. Under the joint constraints of marginal and conditional distribution alignment, the extracted features exhibit improved inter-class discriminability and intra-class consistency, effectively reducing the distribution discrepancy between different domains. Results from multiple transfer tasks on three rotating machinery datasets demonstrate that the proposed method can learn rich transferable features. The overall average accuracy of the proposed method on these three datasets is 88.07%, 89.17%, and 96.63%, respectively, significantly outperforming other methods in terms of robustness and generalization.

Partial transfer learning method based on MDWCAN for rolling bearing fault diagnosis under noisy conditions

Domain adaptation shows excellent diagnostic performance for transfer bearing fault diagnosis (FD) in noise-free scenarios, where the training and test data label spaces are the same. However, in actual engineering, collecting test data for all health states is difficult and the vibration signal is susceptible to noise interference, which limits its application in real industry. Therefore, how to effectively source domain outlier samples and suppress irrelevant noise poses a challenge to the effectiveness of existing FD methods based on domain adaptation. To address the above issues, a novel transfer learning model named multi-scale denoising weighted conditional adversarial network (MDWCAN) is developed for partial transfer FD problem under noisy conditions. Firstly, a global multimodal denoising module is proposed to form a new feature extractor with multi-scale convolutional modules, which can enhance important features and weaken interference features. Secondly, a new loss function is proposed, in which the weight function is introduced into the conditional adversarial function to suppress the negative influence of source domain outlier samples. Finally, experimental results on two bearing datasets show that MDWCAN outperforms other methods.

Electronic Nose Drift Suppression Based on Smooth Conditional Domain Adversarial Networks.

Anti-drift is a new and serious challenge in the field related to gas sensors. Gas sensor drift causes the probability distribution of the measured data to be inconsistent with the probability distribution of the calibrated data, which leads to the failure of the original classification algorithm. In order to make the probability distributions of the drifted data and the regular data consistent, we introduce the Conditional Adversarial Domain Adaptation Network (CDAN)+ Sharpness Aware Minimization (SAM) optimizer-a state-of-the-art deep transfer learning method.The core approach involves the construction of feature extractors and domain discriminators designed to extract shared features from both drift and clean data. These extracted features are subsequently input into a classifier, thereby amplifying the overall model's generalization capabilities. The method boasts three key advantages: (1) Implementation of semi-supervised learning, thereby negating the necessity for labels on drift data. (2) Unlike conventional deep transfer learning methods such as the Domain-adversarial Neural Network (DANN) and Wasserstein Domain-adversarial Neural Network (WDANN), it accommodates inter-class correlations. (3) It exhibits enhanced ease of training and convergence compared to traditional deep transfer learning networks. Through rigorous experimentation on two publicly available datasets, we substantiate the efficiency and effectiveness of our proposed anti-drift methodology when juxtaposed with state-of-the-art techniques.

Cardiac fat segmentation using computed tomography and an image-to-image conditional generative adversarial neural network

In recent years, research has highlighted the association between increased adipose tissue surrounding the human heart and elevated susceptibility to cardiovascular diseases such as atrial fibrillation and coronary heart disease. However, the manual segmentation of these fat deposits has not been widely implemented in clinical practice due to the substantial workload it entails for medical professionals and the associated costs. Consequently, the demand for more precise and time-efficient quantitative analysis has driven the emergence of novel computational methods for fat segmentation. This study presents a novel deep learning-based methodology that offers autonomous segmentation and quantification of two distinct types of cardiac fat deposits. The proposed approach leverages the pix2pix network, a generative conditional adversarial network primarily designed for image-to-image translation tasks. By applying this network architecture, we aim to investigate its efficacy in tackling the specific challenge of cardiac fat segmentation, despite not being originally tailored for this purpose. The two types of fat deposits of interest in this study are referred to as epicardial and mediastinal fats, which are spatially separated by the pericardium. The experimental results demonstrated an average accuracy of 99.08% and f1-score 98.73 for the segmentation of the epicardial fat and 97.90% of accuracy and f1-score of 98.40 for the mediastinal fat. These findings represent the high precision and overlap agreement achieved by the proposed methodology. In comparison to existing studies, our approach exhibited superior performance in terms of f1-score and run time, enabling the images to be segmented in real time.

Ground-motion simulations using two-dimensional convolution condition adversarial neural network(2D-cGAN)

In this paper, an integrated framework based on conditional adversarial neural network (cGAN) is established to simulate ground motions for earthquake scenarios with different magnitude, distance, and local site conditions. A ground motion dataset is created for training the network, within which most data are observed accelerograms collected from PEER and gaps for certain magnitude-site conditions are filled with accelerograms simulated using stochastic finite-fault modeling. During the learning of cGAN, the dataset is classified first with multi-labels according to local site conditions, and the multi-labels are set to be the conditional information Y for the network. Effects of using one-dimensional convolution (Conv1D) and two-dimensional convolution (Conv2D) in cGAN are tested in terms of the time-domain and frequency-domain characteristics of the generated ground motion. The test proves the strengths of Conv2D in capturing high-dimensional temporal and spectral characteristics from one-dimensional time series of ground motion. Ground motions generated using 2D-cGAN network are compared to the measures predicted by widely-used ground motion prediction equations, through which the reliability of 2D-cGAN network on ground motion simulation is verified.

Neural network for full-Stokes image reconstruction with wavelength and modulation independence

In this paper, we propose a full-Stokes image reconstruction (FIR) network to reconstruct polarization images from a snapshot of random modulated image. The Stokes-Mueller matrix theory demonstrates that a image captured by conventional polarization imaging system is a linear superposition of the four Stokes vectors, thus image decomposition methods can be used for full-Stokes image reconstruction. The network is constructed based on a cycled conditional adversarial network, which is trained utilizing the physical mapping relationships between the Stokes vectors and the modulated images. Simulations and experiments are conducted to demonstrate the network’s wavelength independence and modulation independence, proving the effectiveness of proposed FIRnet. To some extent, our method alleviates the contradiction between measurement time and accuracy of polarization, promoting the application of compact snapshot full-Stokes imaging.

Predominant audio source separation in polyphonic music

Predominant source separation is the separation of one or more desired predominant signals, such as voice or leading instruments, from polyphonic music. The proposed work uses time-frequency filtering on predominant source separation and conditional adversarial networks to improve the perceived quality of isolated sounds. The pitch tracks corresponding to the prominent sound sources of the polyphonic music are estimated using a predominant pitch extraction algorithm and a binary mask corresponding to each pitch track and its harmonics are generated. Time-frequency filtering is performed on the spectrogram of the input signal using a binary mask that isolates the dominant sources based on pitch. The perceptual quality of source-separated music signal is enhanced using a CycleGAN-based conditional adversarial network operating on spectrogram images. The proposed work is systematically evaluated using the IRMAS and ADC 2004 datasets. Subjective and objective evaluations have been carried out. The reconstructed spectrogram is converted back to music signals by applying the inverse short-time Fourier transform. The intelligibility of separated audio is enhanced using an intelligibility enhancement module based on an audio style transfer scheme. The performance of the proposed method is compared with state-of-the-art Demucs and Wave-U-Net architectures and shows competing performance both objectively and subjectively.

Using conditional generative adversarial networks for the prediction of stresses in an adhesive composite

AbstractFiber reinforced adhesives are an integral part of wind energy turbine blades and play an important role in evaluation of structural integrity of the blades. Digitization of manufacturing processes and demand for fast and efficient stress analysis argues for the application of deep learning. As a step forward in using deep learning to predict structural properties in composite, the work would explore the applicability of deep learning methods to 2D fiber reinforced composite. In this work, a composite adhesive used in wind energy turbine blades is used as an illustrative example. The composite adhesive when viewed using a CT scan machine shows a heterogeneous structure with two phases, a softer matrix phase and a much stiff fiber phase. For this purpose, several CT scan images of the fiber adhesive composite are used. The images on subsequent FEA analysis serves as the training data for the deep learning framework. Once trained, the neural network would be able to predict the stress distributions in the structure. We apply conditional adversarial networks as general purpose solutions to problems focused on stress prediction, which could be thought of as an image‐to‐image translation from the microstructure domain to the stress domain. These networks in addition to learning the mapping from input image to output image also learn a loss function to train for this mapping. This makes the method more generic because otherwise different loss formulations for different problems have to be formulated.

Generative Target Tracking Method with Improved Generative Adversarial Network

Multitarget tracking is prone to target loss, identity exchange, and jumping problems in the context of complex background, target occlusion, target scale, and pose transformation. In this paper, we proposed a target tracking algorithm based on the conditional adversarial generative twin networks, using the improved you only look once multitarget association algorithm to classify and detect the position of the target to be detected in the current frame, constructing a feature extraction model using generative adversarial networks (GANs) to learn the main features and subtle features of the target, and then using GANs to generate the motion trajectories of multiple targets, finally fuzing the motion and appearance information of the target to obtain the optimal match. The optimal matching of the tracked targets is obtained. The experimental results under OTB2015 and IVOT2018 datasets demonstrate that the proposed multitarget tracking algorithm has high accuracy and robustness, with 65% less jumps and 0.25% more accuracy than the current algorithms with minimal identity exchange and jumps.

ORTHOIMAGE-TO-2D ARCHITECTURAL DRAWING WITH CONDITIONAL ADVERSARIAL NETWORKS

Abstract. Vectorization of orthoimages of Cultural Heritage sites requires a considerable amount of time and constant supervision by qualified professionals. In addition, this 2D architectural drawing creation requires expert knowledge for appropriate interpretation of the orthoimages. In this paper, the use of conditional adversarial networks as a solution to orthoimage-to-drawing translation problems is proposed. The presented work exploits a state of the art conditional Generative Adversarial Network with a Markovian discriminator and modifies it using a ResNet fully convolutional network as generator in order to deliver reliable and accurate 2D architectural drawings in a binary image format. Following the 2D drawing image generation, their automated conversion into vector files is performed through a vectorization function, giving also the possibility to edit and scale the edges. Experimental results over two different Cultural Heritage test sites demonstrates that this approach is highly effective at synthesising 2D architectural drawings from orthoimages in great detail and reliability by learning the interpretation performed by the expert architects during the vectorization process.

Deep learning‐based image processing for financial audit risk quantification in healthcare

AbstractRecently, deep learning technology has gradually penetrated various fields. Today, the field of healthcare is also closely linked to deep learning technology. Image processing technology based on deep learning can accurately segment medical images, which is convenient for medical research and pathological analysis. Accurate distribution of images can effectively save medical resources. Therefore, image processing techniques can contribute to the quantification and assessment of economic audit risks. In recent years, medical image segmentation has achieved many research results. However, with the improvement of accuracy, the segmentation standards of medical images are also becoming more and more stringent. For medical images, they tend to have rough and fuzzy boundaries and noise disturbances of different shapes. The above problems pose challenges for accurate localization and segmentation of lesion regions. On the other hand, in the field of medical images, there are also problems such as unbalanced number of samples and scarcity of large medical image datasets. In response to these problems, this paper conducts research work and proposes an Attention Mechanism and Multi‐Scale spatial Pooling‐based conditional Adversarial Network (AM‐MSP‐cGAN) model to achieve automatic segmentation of medical images. AM‐MSP‐cGAN can learn more detailed features from fuzzy boundaries, and effectively solve the problem of data lack, thereby promoting economic audit risk quantification and assessment in the healthcare field.

Abstract 5402: Using computer vision to resolve proliferative dynamics within therapeutic responses in large scale screens of patient derived models

Abstract IC50, Emax, and AUC values are commonly used to assess susceptibility for a given therapeutic candidate but are subject to the number of cell divisions for a given in vitro model. When comparing multiple in vitro models and especially patient-derived models such as patient-derived tumor organoids, the degree of intrinsic cell proliferation or doubling time substantially differs and therefore limits conventional metrics as above. Furthermore, assays that directly measure cellular viability can oftentimes overlook therapeutic agents that have cytostatic, versus cytotoxic mechanisms of action and require assays with similar sensitivity and precision to account for this phenotype. Finally, terminal endpoint assays can require tedious temporal optimization to capture the full dynamic range of a given therapeutic which makes cross-therapeutic and cross-model comparisons difficult to interpret. An assay that incorporates these multiple features and in vitro specific doubling time would lead to highly accurate measurements of drug effects and allow the calculation of growth-adjusted IC50 values i.e. GI50 values. We, therefore, sought to develop a computer vision model that utilizes label-free longitudinal light microscopy images as input to report the total number of nuclei present within a given experimental well, allowing for the monitoring of cell division and any cytostatic effects. To train the neural network, we use Hoechst-stained images as our ground truth. The model used is an extension of the Regularized Conditional Adversarial Network (PMID: 34320344). It takes as input Brightfield images along with the registered Hoescht stained wells. The label for each well is obtained using commercial image processing software to count the total number of nuclei present in each well from the Hoescht stained channel. The model was trained to predict the virtually stained Hoescht well for every Brightfield, as well as the aggregate nuclei count for all organoids present inside the well. The model was trained on 29000 individual sites across 9 distinct cancer types. Five experiments were conducted during training to test for bit depth and tonality effects of the input training data. Inference on 9240 images showed a pearsonR score of 0.81 on the best-performing model. Results show that the model performs well and can be applied in real-time using only experimental data to correct for differing cell division rates and measure cytostatic phenotypes all while using simple light microscopy. The tool will enable cross-comparison of different therapeutic MoAs as well as enable cross-cancer type/indication comparison for a therapeutic in development to inform early development clinical strategy. Citation Format: Madhavi Kannan, Brian Larsen, Chi-Sing Ho, Jagadish Venkataraman, Martin Stumpe, Ameen Salahudeen. Using computer vision to resolve proliferative dynamics within therapeutic responses in large scale screens of patient derived models. [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2023; Part 1 (Regular and Invited Abstracts); 2023 Apr 14-19; Orlando, FL. Philadelphia (PA): AACR; Cancer Res 2023;83(7_Suppl):Abstract nr 5402.

Retrieving Vertical Cloud Radar Reflectivity from MODIS Cloud Products with CGAN: An Evaluation for Different Cloud Types and Latitudes

Retrieving cloud vertical structures with satellite remote-sensing measurements is highly desirable and technically challenging. In this paper, the conditional adversarial neural network (CGAN) for retrieving the equivalent cloud radar reflectivity at 94 GHz of the Cloud Profile Radar (CPR) onboard CloudSat is extended and evaluated comprehensively for different cloud types and geographical regions. The CGAN-based retrieval model was extended with additional data samples and improved with a new normalization adjustment. The model was trained with the labeled datasets of the moderate-resolution imaging spectroradiometer (MODIS) cloud top pressure, cloud water path, cloud optical thickness, and effective particle radius data, and the CloudSat/CPR reflectivity from 2010 to 2017 over the global oceans. The test dataset, containing 24,427 cloud samples, was statistically analyzed to assess the performance of the model for eight cloud types and three latitude zones with multiple verification metrics. The results show that the CGAN model possesses good reliability for retrieving clouds with reflectivity > −25 dBZ. The model performed the best for deep convective systems, followed by nimbostratus, altostratus, and cumulus, but presented a very limited ability for stratus, cirrus, and altocumulus. The model performs better in the low and middle latitudes than in the high latitudes. This work demonstrated that the CGAN model can be used to retrieve vertical structures of deep convective clouds and nimbostratus with great confidence in the mid- and lower latitude region, laying the ground for retrieving reliable 3D cloud structures of the deep convective systems including convective storms and hurricanes from MODIS cloud products and used for predicting these storms.

MRI image synthesis for fluid-attenuated inversion recovery and diffusion-weighted images with deep learning.

This study aims to synthesize fluid-attenuated inversion recovery (FLAIR) and diffusion-weighted images (DWI) with a deep conditional adversarial network from T1- and T2-weighted magnetic resonance imaging (MRI) images. A total of 1980 images of 102 patients were split into two datasets: 1470 (68 patients) in a training set and 510 (34 patients) in a test set. The prediction framework was based on a convolutional neural network with a generator and discriminator. T1-weighted, T2-weighted, and composite images were used as inputs. The digital imaging and communications in medicine (DICOM) images were converted to 8-bit red-green-blue images. The red and blue channels of the composite images were assigned to 8-bit grayscale pixel values in T1-weighted images, and the green channel was assigned to those in T2-weighted images. The prediction FLAIR and DWI images were of the same objects as the inputs. For the results, the prediction model with composite MRI input images in the DWI image showed the smallest relative mean absolute error (rMAE) and largest mutual information (MI), and that in the FLAIR image showed the largest relative mean-square error (rMSE), relative root-mean-square error (rRMSE), and peak signal-to-noise ratio (PSNR). For the FLAIR image, the prediction model with the T2-weighted MRI input images generated more accurate synthesis results than that with the T1-weighted inputs. The proposed image synthesis framework can improve the versatility and quality of multi-contrast MRI without extra scans. The composite input MRI image contributes to synthesizing the multi-contrast MRI image efficiently.

Image-to-image translation for wavefront and point spread function estimation

We develop and evaluate a new approach to phase estimation for observational astronomy that can be used for accurate point spread function reconstruction. Phase estimation is required where a terrestrial observatory uses an adaptive optics (AO) system to assist astronomers in acquiring sharp, high-contrast images of faint and distant objects. Our approach is to train a conditional adversarial artificial neural network architecture to predict phase using the wavefront sensor data from a closed-loop AO system. We present a detailed simulation study under different turbulent conditions, using the retrieved residual phase to obtain the point spread function of the simulated instrument. Compared to the state-of-the-art model-based approach in astronomy, our approach is not explicitly limited by modeling assumptions, e.g., independence between terms, such as bandwidth and anisoplanatism—and is conceptually simple and flexible. We use the open-source COMPASS tool for end-to-end simulations. On key quality metrics, specifically the Strehl ratio and Halo distribution in our application domain, our approach achieves results better than the model-based baseline.

Domain Adaptive Graph Infomax via Conditional Adversarial Networks

The emerging graph neural networks (GNNs) have demonstrated impressive performance on the node classification problem in complex networks. However, existing GNNs are mainly devised to classify nodes in a (partially) labeled graph. To classify nodes in a newly-collected unlabeled graph, it is desirable to transfer label information from an existing labeled graph. To address this cross-graph node classification problem, we propose a graph infomax method that is domain adaptive. Node representations are computed through neighborhood aggregation. Mutual information is maximized between node representations and global summaries, encouraging node representations to encode the global structural information. Conditional adversarial networks are employed to reduce the domain discrepancy by aligning the multimodal distributions of node representations. Experimental results in real-world datasets validate the performance of our method in comparison with the state-of-the-art baselines.

Approximating global illumination with ambient occlusion and environment light via generative adversarial networks

Calculating global illumination in computer graphics is difficult, especially for complex scenes. This is due to the interreflections of the rays of light and the interactions with the materials of the objects composing the scene. Solutions based on ambient light approximations have been implemented. However, these are computationally intensive and produce images with less precision, as these solutions ignore the ambient lighting component by adopting coarse approximations. In this paper, we propose a method capable of approximating the global illumination effect. Our idea is to compute the global illumination by adding three images (direct illumination, environmental light, and ambient occlusion). Direct illumination is calculated by a reference method. Environmental illumination is computed using an adversarial neural network from a single 2D image. Ambient occlusion is generated using conditional adversarial neural networks with an attention mechanism to pay more attention to the relevant image features during the training step. We use two image masks to keep the object’s position in the screen space, which allows efficient reconstruction of the final result. Our solution produces quality images compared to reference images and does not require any computation in the 3D scene or screen space.