Generative Adversarial Network Framework Research Articles

MCS-GAN: A Different Understanding for Generalization of Deep Forgery Detection

After several years of development, deep synthesis technology has made significant progress in image and video synthesis. Deep forgery represented by Deepfakes has become a research hotspot, which is used as a tool for disinformation attacks. The current strongly discriminative models can have good performance on specific datasets, even close to <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$100\%$</tex-math></inline-formula> accuracy. Unfortunately, since a specific discriminative method only fits a specific data distribution, and different forgery methods or datasets have different data distributions. These methods fail to achieve high performance in cross-dataset detection. In response to this problem and focusing on the actual situation, we adjust the strong generalization detection across the dataset to the generalization detection of unseen fake video. We propose Multi-Crise-Cross Attention and StyleGANv2 Generative Adversarial Network (MCS-GAN). Firstly, we built a Generative Adversarial Network (GAN) framework to learn the distribution of real face data and generate corresponding face images. Secondly, to break the high stitch between the fake region and the background, the model needs to have strong enough feature analysis and pixel restoration capabilities. Therefore, we propose a generator consisting of a Multi-Crise-Cross-Attention (MC) encoder and a StyleGANv2 (SG2) decoder. Finally, to avoid the situation where as long as a face is normal or different faces are abnormal, we set a latent space encoding discriminator and increase the ratio of latent space vector, so as to detect anomaly generated by the forgery operation acting on latent space. We conduct some model generalization experiments on videos on the Internet and some popular deepfake databases. The results show that the accuracy of our method is better compared with the best methods.

Scrupulous SCGAN Framework for Recognition of Restored Images with Caffe based PCA Filtration

Computer vision enables to detect many objects in any scenario which helps in various real time application but still face recognition and detection remains a tedious process due to the low resolution, blurriness, noise, diverse pose and expression and occlusions. This proposal develops a novel scrupulous Standardized Convolute Generative Adversarial Network (SCGAN) framework for performing accurate face recognition automatically by restoring the occluded region including blind face restoration. Initially, a scrupulous image refining technique is utilised to offer the appropriate input to the network in the subsequent process. Following the pre-processing stage, a Caffe based Principle Component Analysis (PCA) filtration is conducted which uses convolutional architecture for fast feature embedding that collects spatial information and significant differentiating characteristics to counteract the loss of information existing in pooling operations. Then a filtration method identifies the specific match of the face based on the extracted features, creating uncorrelated variables that optimise variance across time while minimising information loss. To handle all the diversification occurring in the image and accurately recognise the face with occlusion in any part of the face, a novel Standardized Convolute GAN network is used to restore the image and recognise the face using novel Generative Adversarial Network (GAN) networks are modelled. This GAN ensures the normal distribution along with parametric optimization contributing to the high performance with accuracy of 96.05% and Peak Signal to Noise Ratio (PSNR) of 18 and Structural Similarity Index Metric (SSIM) of 98% for restored face recognition. Thus, the performance of the framework based on properly recognizing the face from the generated images is evaluated and discussed.

Scrupulous SCGAN Framework for Recognition of Restored Images with Caffe based PCA Filtration

Computer vision enables to detect many objects in any scenario which helps in various real time application but still face recognition and detection remains a tedious process due to the low resolution, blurriness, noise, diverse pose and expression and occlusions. This proposal develops a novel scrupulous Standardized Convolute Generative Adversarial Network (SCGAN) framework for performing accurate face recognition automatically by restoring the occluded region including blind face restoration. Initially, a scrupulous image refining technique is utilised to offer the appropriate input to the network in the subsequent process. Following the pre-processing stage, a Caffe based Principle Component Analysis (PCA) filtration is conducted which uses convolutional architecture for fast feature embedding that collects spatial information and significant differentiating characteristics to counteract the loss of information existing in pooling operations. Then a filtration method identifies the specific match of the face based on the extracted features, creating uncorrelated variables that optimise variance across time while minimising information loss. To handle all the diversification occurring in the image and accurately recognise the face with occlusion in any part of the face, a novel Standardized Convolute GAN network is used to restore the image and recognise the face using novel Generative Adversarial Network (GAN) networks are modelled. This GAN ensures the normal distribution along with parametric optimization contributing to the high performance with accuracy of 96.05% and Peak Signal to Noise Ratio (PSNR) of 18 and Structural Similarity Index Metric (SSIM) of 98% for restored face recognition. Thus, the performance of the framework based on properly recognizing the face from the generated images is evaluated and discussed.

Modular GAN: positron emission tomography image reconstruction using two generative adversarial networks.

The reconstruction of PET images involves converting sinograms, which represent the measured counts of radioactive emissions using detector rings encircling the patient, into meaningful images. However, the quality of PET data acquisition is impacted by physical factors, photon count statistics and detector characteristics, which affect the signal-to-noise ratio, resolution and quantitative accuracy of the resulting images. To address these influences, correction methods have been developed to mitigate each of these issues separately. Recently, generative adversarial networks (GANs) based on machine learning have shown promise in learning the complex mapping between acquired PET data and reconstructed tomographic images. This study aims to investigate the properties of training images that contribute to GAN performance when non-clinical images are used for training. Additionally, we describe a method to correct common PET imaging artefacts without relying on patient-specific anatomical images. The modular GAN framework includes two GANs. Module 1, resembling Pix2pix architecture, is trained on non-clinical sinogram-image pairs. Training data are optimised by considering image properties defined by metrics. The second module utilises adaptive instance normalisation and style embedding to enhance the quality of images from Module 1. Additional perceptual and patch-based loss functions are employed in training both modules. The performance of the new framework was compared with that of existing methods, (filtered backprojection (FBP) and ordered subset expectation maximisation (OSEM) without and with point spread function (OSEM-PSF)) with respect to correction for attenuation, patient motion and noise in simulated, NEMA phantom and human imaging data. Evaluation metrics included structural similarity (SSIM), peak-signal-to-noise ratio (PSNR), relative root mean squared error (rRMSE) for simulated data, and contrast-to-noise ratio (CNR) for NEMA phantom and human data. For simulated test data, the performance of the proposed framework was both qualitatively and quantitatively superior to that of FBP and OSEM. In the presence of noise, Module 1 generated images with a SSIM of 0.48 and higher. These images exhibited coarse structures that were subsequently refined by Module 2, yielding images with an SSIM higher than 0.71 (at least 22% higher than OSEM). The proposed method was robust against noise and motion. For NEMA phantoms, it achieved higher CNR values than OSEM. For human images, the CNR in brain regions was significantly higher than that of FBP and OSEM (p < 0.05, paired t-test). The CNR of images reconstructed with OSEM-PSF was similar to those reconstructed using the proposed method. The proposed image reconstruction method can produce PET images with artefact correction.

Investigating Methods of Searching for Key Frames in Video Flow with the Use of Neural Networks for Search Systems

Various methods of video content data analysis are presented, compared, and evaluated in this paper. Due to the analysis, the most effective strategies for video data processing involve searching for key frames within the video stream. The examined methods are categorized into consistent comparison, global comparison based on clustering, and event/object-based methodologies. Key techniques such as sequence search, classification, frame decoding, and anomaly detection are singled out as particularly valuable for comparison and matching tasks. The research further reveals that artificial intelligence and machine learning-driven methods reign supreme in this domain, with deep learning approaches outperforming traditional techniques. The employment of convolutional neural networks and attention mechanisms to capture the temporal intricacies across variable scopes is especially noteworthy. Additionally, leveraging the Actor-Critic model within a Generative Adversarial Network framework has shown encouraging outcomes. A significant highlight of the study is the proposed approach which incorporates modified Independent Recurrent Neural Networks (IndRNN) complemented by an attention mechanism. The enhancement using mathematical tools, notably the standard deviation, for key frame detection, exemplifies the potential of integrating analytical instruments to refine the system's precision. Such advancements, as presented in this research, pave the way for substantial enhancements in information systems tailored for video content analysis and source identification.

A Method of Enhancing Silk Digital Printing Color Prediction through Pix2Pix GAN-Based Approaches

Color prediction and color management for digital printed fabrics remain a challenging task. Accurate prediction of color appearances of digital printed fabrics would enable designers and manufacturers to better fulfill their design requirements and creative visions. We propose a color prediction method for silk digital printing utilizing a Pix2Pix Generative Adversarial Network (GAN) framework. This method aims to generate predicted images that possess the same stylistic and color characteristics as their actual fabrics after production. To develop and validate the method, color data and images are collected and processed from 5252 sets of paired original Pantone TPX color card and actual print sample fabrics. The results of this study demonstrate that the method can predict the colors of silk digital print samples while effectively reproducing the effects of inkjet printing in silk fabrics including silk crepe satin and silk twill. The method exhibits high prediction accuracy to an average CIEDE2000 value of 2.372 for silk crepe satin and 1.846 for silk twill. The findings of this research not only enhance the efficiency and accuracy of color management in fabric digital printing technology but also contribute to the exploration and development of high-fidelity color prediction techniques within the textile industry.

PLM-GAN: A Large-Scale Protein Loop Modeling Using pix2pix GAN.

Revealing the tertiary structure of proteins holds huge significance as it unveils their vital properties and functions. These intricate three-dimensional configurations comprise diverse interactions including ionic, hydrophobic, and disulfide forces. In certain instances, these structures exhibit missing regions, necessitating the reconstruction of specific segments, thereby resulting in challenges in protein design, which encompasses loop modeling, circular permutation, and interface prediction. To address this problem, we present two pioneering models: pix2pix generative adversarial network (GAN) and PLM-GAN. The pix2pix GAN model is adept at generating and inpainting distance matrices of protein structures, whereas the PLM-GAN model incorporates residual blocks into the U-Net network of the GAN, building upon the foundation of the pix2pix GAN model. To bolster the models' performance, we introduce a novel loss function named the "missing to real regions loss" (LMTR) within the GAN framework. Additionally, we introduce a distinctive approach of pairing two different distance matrices: one representing the native protein structure and the other representing the same structure with a missing region that undergoes changes in each successive epoch. Moreover, we extend the reconstruction of missing regions, encompassing up to 30 amino acids and increase the protein length by 128 amino acids. The evaluation of our pix2pix GAN and PLM-GAN models on a random selection of natural proteins (4ZCB, 3FJB, and 2REZ) demonstrated promising experimental results. Our models constitute significant contributions to addressing intricate challenges in protein structure design. These contributions hold immense potential to propel advancements in protein-protein interactions, drug design, and further innovations in protein engineering. Data, code, trained models, examples, and measurements are available on https://github.com/mena01/PLM-GAN-A-Large-Scale-Protein-Loop-Modeling-Using-pix2pix-GAN_.

Measurement of solid size in early-stage lung adenocarcinoma by virtual 3D thin-section CT applied artificial intelligence

An artificial intelligence (AI) system that reconstructs virtual 3D thin-section CT (TSCT) images from conventional CT images by applying deep learning was developed. The aim of this study was to investigate whether virtual and real TSCT could measure the solid size of early-stage lung adenocarcinoma. The pair of original thin-CT and simulated thick-CT from the training data with TSCT images (thickness, 0.5–1.0 mm) of 2700 pulmonary nodules were used to train the thin-CT generator in the generative adversarial network (GAN) framework and develop a virtual TSCT AI system. For validation, CT images of 93 stage 0–I lung adenocarcinomas were collected, and virtual TSCTs were reconstructed from conventional 5-mm thick-CT images using the AI system. Two radiologists measured and compared the solid size of tumors on conventional CT and virtual and real TSCT. The agreement between the two observers showed an almost perfect agreement on the virtual TSCT for solid size measurements (intraclass correlation coefficient = 0.967, P < 0.001, respectively). The virtual TSCT had a significantly stronger correlation than that of conventional CT (P = 0.003 and P = 0.001, respectively). The degree of agreement between the clinical T stage determined by virtual TSCT and the clinical T stage determined by real TSCT was excellent in both observers (k = 0.882 and k = 0.881, respectively). The AI system developed in this study was able to measure the solid size of early-stage lung adenocarcinoma on virtual TSCT as well as on real TSCT.

Unveiling the future of breast cancer assessment: a critical review on generative adversarial networks in elastography ultrasound.

Elastography Ultrasound provides elasticity information of the tissues, which is crucial for understanding the density and texture, allowing for the diagnosis of different medical conditions such as fibrosis and cancer. In the current medical imaging scenario, elastograms for B-mode Ultrasound are restricted to well-equipped hospitals, making the modality unavailable for pocket ultrasound. To highlight the recent progress in elastogram synthesis, this article performs a critical review of generative adversarial network (GAN) methodology for elastogram generation from B-mode Ultrasound images. Along with a brief overview of cutting-edge medical image synthesis, the article highlights the contribution of the GAN framework in light of its impact and thoroughly analyzes the results to validate whether the existing challenges have been effectively addressed. Specifically, This article highlights that GANs can successfully generate accurate elastograms for deep-seated breast tumors (without having artifacts) and improve diagnostic effectiveness for pocket US. Furthermore, the results of the GAN framework are thoroughly analyzed by considering the quantitative metrics, visual evaluations, and cancer diagnostic accuracy. Finally, essential unaddressed challenges that lie at the intersection of elastography and GANs are presented, and a few future directions are shared for the elastogram synthesis research.

Novel hybrid generative adversarial network for synthesizing image from sketch

In the area of sketch-based image retrieval process, there is a potential difference between retrieving the match images from defined dataset and constructing the synthesized image. The former process is quite easier while the latter process requires more faster, accurate, and intellectual decision making by the processor. After reviewing open-end research problems from existing approaches, the proposed scheme introduces a computational framework of hybrid generative adversarial network (GAN) as a solution to address the identified research problem. The model takes the input of query image which is processed by generator module running 3 different deep learning modes of ResNet, MobileNet, and U-Net. The discriminator module processes the input of real images as well as output from generator. With a novel interactive communication between generator and discriminator, the proposed model offers optimal retrieval performance along with an inclusion of optimizer. The study outcome shows significant performance improvement.

Deep Learning Based Estimation of Synthetic Tau PET from Amyloid PET

AbstractBackgroundAlzheimer’s disease (AD) is the most common pathological substrate for dementia. Both tau and amyloid positron emission tomography (PET) scans are valuable for evaluating innovative treatments and studying AD progression. However, tau‐PET is extremely expensive and not universally available. Estimating tau‐PET images from amyloid‐PET would mitigate these concerns. In this study, we proposed a novel application of the Conditional Generative Adversarial Network (cGAN) framework to generate high quality tau‐PET (18F‐flortaucipir) scans from the corresponding amyloid‐PET (18F‐florbetapir) scans.MethodA dataset of 475 individuals who underwent both 18F‐florbetapir and 18F‐flortaucipir scans with different clinical diagnosis (Table 1) were used in our study from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) online depository. Our cGAN model was trained for 50 epochs using real, corresponding amyloid‐tau PET pairs. The resulting model takes real amyloid‐PET images as input and outputs synthetic tau‐PET images (Figure 1.a). The similarity between synthetic and real tau‐PET images was evaluated via the structural similarity index (SSIM), peak signal to noise ratio (PSNR), and normalized root mean squared error (NRMSE). Furthermore, we extracted regional standard uptake value ratios (SUVRs) and meta‐region of interest (ROI) SUVRs from the real tau‐PET scans and synthetic tau‐PET scans. The utility of the synthetic tau‐PET meta‐ROI SUVR for identifying real tau‐PET scans that are tau positive was evaluated in terms of true positive rate (TPR), false positive rate (FPR), and the area under the receiver operating characteristic (ROC) curve (AUC).ResultOur cGAN model yielded a SSIM of 0.917, a PSNR of 27.04, and a NRSME of 0.18 on the test set. The ROC curve analysis (Figure 2) resulted in an AUC of 0.84%. By applying a cut‐off threshold corresponding to the model optimal operating point, a TPR of 90% and FPR of 28% were achieved. The SUVR analysis yielded a mean absolute percentage error of 8.3% ± 6.5% on the test dataset.ConclusionThe proposed cGAN model generated high quality 18F‐flortaucipir PET images from 18F‐florbetapir PET images (Figure 1.b). This could be highly useful in a screening paradigm to use amyloid‐PET to identify those individuals most likely to be tau positive.

Synthetic 3D Spinal Vertebrae Reconstruction from Biplanar X-rays Utilizing Generative Adversarial Networks.

Computed tomography (CT) offers detailed insights into the internal anatomy of patients, particularly for spinal vertebrae examination. However, CT scans are associated with higher radiation exposure and cost compared to conventional X-ray imaging. In this study, we applied a Generative Adversarial Network (GAN) framework to reconstruct 3D spinal vertebrae structures from synthetic biplanar X-ray images, specifically focusing on anterior and lateral views. The synthetic X-ray images were generated using the DRRGenerator module in 3D Slicer by incorporating segmentations of spinal vertebrae in CT scans for the region of interest. This approach leverages a novel feature fusion technique based on X2CT-GAN to combine information from both views and employs a combination of mean squared error (MSE) loss and adversarial loss to train the generator, resulting in high-quality synthetic 3D spinal vertebrae CTs. A total of n = 440 CT data were processed. We evaluated the performance of our model using multiple metrics, including mean absolute error (MAE) (for each slice of the 3D volume (MAE0) and for the entire 3D volume (MAE)), cosine similarity, peak signal-to-noise ratio (PSNR), 3D peak signal-to-noise ratio (PSNR-3D), and structural similarity index (SSIM). The average PSNR was 28.394 dB, PSNR-3D was 27.432, SSIM was 0.468, cosine similarity was 0.484, MAE0 was 0.034, and MAE was 85.359. The results demonstrated the effectiveness of this approach in reconstructing 3D spinal vertebrae structures from biplanar X-rays, although some limitations in accurately capturing the fine bone structures and maintaining the precise morphology of the vertebrae were present. This technique has the potential to enhance the diagnostic capabilities of low-cost X-ray machines while reducing radiation exposure and cost associated with CT scans, paving the way for future applications in spinal imaging and diagnosis.

Model-Informed Generative Adversarial Network (MI-GAN) for Learning Optimal Power Flow

The optimal power flow (OPF) problem, as a critical component of power system operations, becomes increasingly difficult to solve due to the variability, intermittency, and unpredictability of renewable energy brought to the power system. Although traditional optimization techniques, such as stochastic and robust optimization approaches, could be leveraged to address the OPF problem, in the face of renewable energy uncertainty, i.e., the dynamic coefficients in the optimization model, their effectiveness in dealing with large-scale problems remains limited. As a result, deep learning techniques, such as neural networks, have recently been developed to improve computational efficiency in solving OPF problems with the utilization of data. However, the feasibility and optimality of the solution may not be guaranteed, and the system dynamics cannot be properly addressed as well. In this paper, we propose an optimization model-informed generative adversarial network (MI-GAN) framework to solve OPF under uncertainty. The main contributions are summarized into three aspects: (1) to ensure feasibility and improve optimality of generated solutions, three important layers are proposed: feasibility filter layer, comparison layer, and gradient-guided layer; (2) in the GAN-based framework, an efficient model-informed selector incorporating these three new layers is established; and (3) a new recursive iteration algorithm is also proposed to improve solution optimality and handle the system dynamics. The numerical results on IEEE test systems show that the proposed method is very effective and promising.

A Generative Adversarial and Spatiotemporal Differential Fusion Method in Radar Echo Extrapolation

As an important part of remote sensing data, weather radar plays an important role in convective weather forecasts to reduce extreme precipitation disasters. The existing radar echo extrapolation methods do not utilize the local natural characteristics of the radar echo effectively but only roughly extract the whole characteristics of the radar echo. To address these challenges, we design a spatiotemporal difference and generative adversarial fusion model (STDGAN). Specifically, a spatiotemporal difference module (STD) is designed to extract local weather patterns and model them in detail. In our model, spatiotemporal difference information and spatiotemporal features captured by the model itself are fused together. In addition, our model is trained in a generative adversarial network (GAN) framework; it helps to generate a clearer map of future radar echoes at the image level. The discriminator consists of multi-scale feature extractors, which can simulate weather models of various scales more completely. Finally, extrapolation experiments were conducted using actual radar echo data from Shijiazhuang and Nanjing. The experiments have shown that our model has a more accurate prediction performance for predicting local weather patterns and overall echo change trajectories compared with previous research models. Our model achieved MSE, PSNE, and SSIM values of 132.22, 37.87, and 0.796, respectively, on the Shijiazhuang radar echo dataset. In addition, our model also showed better performance results on the Nanjing radar echo dataset. The results show that the MSE was 49.570, the PSNR was 0.714, and the SSIM was 30.633. The CC value was 0.855.

Modified query expansion through generative adversarial networks for information extraction in e-commerce

This work addresses an alternative approach for query expansion (QE) using a generative adversarial network (GAN) to enhance the effectiveness of information search in e-commerce. We propose a modified QE conditional GAN (mQE-CGAN) framework, which resolves keywords by expanding the query with a synthetically generated query that proposes semantic information from text input. we train a sequence-to-sequence transformer model as the generator to produce keywords and use a recurrent neural network model as the discriminator to classify an adversarial output with the generator. with the modified CGAN framework, Various forms of semantic insights gathered from the query-document corpus are introduced to the generation process. We leverage these insights as conditions for the generator model and discuss their effectiveness for the query expansion task. our experiments demonstrate that the utilization of condition structures within the mQE-CGAN framework can increase the semantic similarity between generated sequences and reference documents up to nearly 10% compared to baseline models.

Reconstruction-based anomaly detection for multivariate time series using contrastive generative adversarial networks

The majority of existing anomaly detection methods for multivariate time series are based on Transformers and Autoencoders owing to their superior capabilities. However, these methods are susceptible to overfitting when there is insufficient data. To address this issue, we propose a novel unsupervised anomaly detection framework, which seamlessly integrates contrastive learning and Generative Adversarial Networks. More concretely, we utilize data augmentation techniques that incorporate geometric distribution masks to expand our training data, thereby enhancing its diversity. Then, a Transformer-based Autoencoder is trained within a Generative Adversarial Network framework to capture the underlying distribution for normal points. Additionally, we incorporate a contrastive loss into our discriminator to effectively regulate the GAN and ensure good generalization. Finally, anomalies are detected based on reconstruction errors. Numerous experiments on five real-world datasets demonstrated that our proposed method can effectively mitigates overfitting issues and obtains superior performance compared to state-of-the-art approaches. In particular, our model could achieve an average improvement of 9.28% in Precision, 11.33% in Recall, and 11.73% in F1-score.

Implementation of Synthesize GAN Model to Detect Outlier in National Stock Exchange Time Series Multivariate Data

This research work explores a novel approach for identifying outliers in stock related time series multivariate datasets, using Generative Adversarial Networks (GANs). The proposed framework harnesses the power of GANs to create synthetic data points that replicate the statistical characteristics of genuine stock related time series. The use of Generative Adversarial Networks to generate tabular data has become more important in a number of industries, including banking, healthcare, and data privacy. The process of synthesizing tabular data with GANs is also provided in this paper. It involves several critical steps, including data collection, preprocessing, and exploration, as well as the design and training using Generator and Discriminator networks. While the discriminator separates genuine samples from synthetic ones, the generator is in charge of producing synthetic data. Generating high quality tabular data with GANs is a complex task, but it has the potential to facilitate data generation in various domains while preserving data privacy and integrity. The results from the experiments confirm that the GAN framework is useful for detecting outliers. The model demonstrates its proficiency in identifying outliers within stock-related time series data. In comparison, our proposed work also examines the statistics and machine learning models in related application fields.

Data augmentation for design of concentric tube continuum robots by generative adversarial networks

AbstractConcentric tube continuum robots are a promising type of robot for various medical applications. Their application in neurosurgery poses challenging requirements for design and control that can be addressed by physics‐informed data‐based approaches. A prerequisite to data‐based modeling is an informative, rich data set. However, limited access to experimental data raises interest in partially or entirely synthetic data sets. In this contribution, we study the application of generative adversarial networks (GANs) for data augmentation in a data‐based design process of such robots. We propose a GAN framework suitable for curve‐fitting to generate synthetic trajectories of robots along with their corresponding control parameters. Our evaluation shows that the GANs can efficiently produce meaningful synthetic trajectories and control parameter pairs that show a good agreement with simulated trajectories.

2S-BUSGAN: A Novel Generative Adversarial Network for Realistic Breast Ultrasound Image with Corresponding Tumor Contour Based on Small Datasets.

Deep learning (DL) models in breast ultrasound (BUS) image analysis face challenges with data imbalance and limited atypical tumor samples. Generative Adversarial Networks (GAN) address these challenges by providing efficient data augmentation for small datasets. However, current GAN approaches fail to capture the structural features of BUS and generated images lack structural legitimacy and are unrealistic. Furthermore, generated images require manual annotation for different downstream tasks before they can be used. Therefore, we propose a two-stage GAN framework, 2s-BUSGAN, for generating annotated BUS images. It consists of the Mask Generation Stage (MGS) and the Image Generation Stage (IGS), generating benign and malignant BUS images using corresponding tumor contours. Moreover, we employ a Feature-Matching Loss (FML) to enhance the quality of generated images and utilize a Differential Augmentation Module (DAM) to improve GAN performance on small datasets. We conduct experiments on two datasets, BUSI and Collected. Moreover, results indicate that the quality of generated images is improved compared with traditional GAN methods. Additionally, our generated images underwent evaluation by ultrasound experts, demonstrating the possibility of deceiving doctors. A comparative evaluation showed that our method also outperforms traditional GAN methods when applied to training segmentation and classification models. Our method achieved a classification accuracy of 69% and 85.7% on two datasets, respectively, which is about 3% and 2% higher than that of the traditional augmentation model. The segmentation model trained using the 2s-BUSGAN augmented datasets achieved DICE scores of 75% and 73% on the two datasets, respectively, which were higher than the traditional augmentation methods. Our research tackles imbalanced and limited BUS image data challenges. Our 2s-BUSGAN augmentation method holds potential for enhancing deep learning model performance in the field.

Two Birds with One Stone: Boosting Code Generation and Code Search via a Generative Adversarial Network

Automatically transforming developers' natural language descriptions into source code has been a longstanding goal in software engineering research. Two types of approaches have been proposed in the literature to achieve this: code generation, which involves generating a new code snippet, and code search, which involves reusing existing code. However, despite existing efforts, the effectiveness of the state-of-the-art techniques remains limited. To seek for further advancement, our insight is that code generation and code search can help overcome the limitation of each other: the code generator can benefit from feedback on the quality of its generated code, which can be provided by the code searcher, while the code searcher can benefit from the additional training data augmented by the code generator to better understand code semantics. Drawing on this insight, we propose a novel approach that combines code generation and code search techniques using a generative adversarial network (GAN), enabling mutual improvement through the adversarial training. Specifically, we treat code generation and code search as the generator and discriminator in the GAN framework, respectively, and incorporate several customized designs for our tasks. We evaluate our approach in eight different settings, and consistently observe significant performance improvements for both code generation and code search. For instance, when using NatGen, a state-of-the-art code generator, as the generator and GraphCodeBERT, a state-of-the-art code searcher, as the discriminator, we achieve a 32% increase in CodeBLEU score for code generation, and a 12% increase in mean reciprocal rank for code search on a large-scale Python dataset, compared to their original performances.