Generative Network Research Articles

Image fusion using Y-net-based extractor and global-local discriminator

Although some deep learning-based image fusion approaches have realized promising results, how to extract information-rich features from different source images while preserving them in the fused image with less distortions remains challenging issue that needs to be addressed. Here, we propose a well worked-out GAN-based scheme with multi-scale feature extractor and global-local discriminator for infrared and visible image fusion. We use Y-Net as the backbone architecture to design the generator network, and introduce the residual dense block (RDblock) to yield more realistic fused images for infrared and visible images by learning discriminative multi-scale representations that are closer to the essence of different modal images. During feature reconstruction, the cross-modality shortcuts with contextual attention (CMSCA) are employed to selectively aggregate features at different scales and different levels to construct information-rich fused images with better visual effect. To ameliorate the information content of the fused image, we not only constrain the structure and contrast information using structural similarity index, but also evaluate the intensity and gradient similarities at both feature and image levels. Two global-local discriminators that combine global GAN with PatchGAN as a unified architecture help to dig for finer differences between the generated image and reference images, which force the generator to learn both the local radiation information and pervasive global details in two source images. It is worth mentioning that image fusion is achieved during confrontation without fusion rules. Lots of assessment tests demonstrate that the reported fusion scheme achieves superior performance against state-of-the-art works in meaningful information preservation.

A Digital Animation Generation Model Based on Cycle Adversarial Neural Network

This paper proposes a digital animation generation model based on Cycle Adversarial Neural Network (CycleGAN). Compared with the classical CycleGAN, this research presents a multi-attention approach to enhance the network’s generalization. Specifically, a style enhancement module and a style cross-attention mechanism are introduced into the generator network structure, which enables the model to better parse the structural information of the content image and realize the accurate matching of content features and style features. Furthermore, the introduction of a multi-scale discriminator with a fused attention mechanism enhances the preservation of content information from the source image in the output animated image. The experiments conducted in this study demonstrate that the model proposed in this paper exhibits superior performance in generating digital animation. This model not only enhances the realism and variety of the generated effects, but also achieves significant improvements in the coherence and stability of long time-series animation. The contribution of this paper lies in the introduction of the multi-attention mechanism, which enhances the generalization of the network and is of great theoretical and practical significance for the development of the field of digital animation generation.

Ontoloji-Based Generalized Zero-Shot Learning with Generative Networks

Ontoloji-Based Generalized Zero-Shot Learning with Generative Networks

Real-time masked face recognition using deep learning-based double generator network

Real-time masked face recognition using deep learning-based double generator network

Image Generation from Text

Abstract: This work diverts from the usual by using Generative Adversarial Networks (GANs) to intentionally create flawed yet captivating images. We introduce a controlled chaos system, where a stalling generator network and a disoriented discriminator network co-exist. This strange training method encourages the development of peculiar irregularities within semi-realistic images. We welcome the unpredictability that arises from this chaos, purposely avoiding issues like mode collapse. Moreover, the project delves into a new evaluation paradigm for evaluating these deliberately flawed images. We go beyond typical metrics, exploring the use of faulty metrics and questionable visual inspection techniques. In addition, we investigate the impact of clumsy dataset selection, potentially employing unconventional or tampered datasets to train the GAN for specific types of errors. This work explores the potential of GANs for jumbled up art creation, data manipulation, and image confusion, opening a new frontier in GAN applications. While potentially causing confusion in the field, this research also reveals the unimaginable artistic and data exploration capabilities of GANs.

Self-clustered GAN for precipitation nowcasting

This paper proposes a novel GAN framework with self-clustering approach for precipitation nowcasting (ClusterCast). Previous studies have primarily captured the motion vector using only a single latent space, making the models difficult to adapt to disparate space-time distribution of precipitation. Environmental factors (e.g., regional characteristics and precipitation scale) have an impact on precipitation systems and can cause non-stationary distribution. To tackle this problem, our key idea is to train a generator network to predict future radar frames by learning a sub-network that automatically labels precipitation types from a generative model. The training process consists of (i) clustering the hierarchical features derived from the generator stem using a sub-network and (ii) predicting future radar frames according to the self-supervised labels, enabling heterogeneous latent representation. Additionally, we attempt an ensemble forecast that prescribes random perturbations to improve performance. With the flexibility of representation learning, ClusterCast enables the model to learn precipitation distribution more accurately. Results indicate that our method generates non-blurry future frames by preventing mode collapse, and the proposed method demonstrates robustness across various precipitation scenarios. Extensive experiments demonstrate that our method outperforms four benchmarks on a 2-h prediction basis with a mean squared error (MSE) of 8.9% on unseen datasets.

Automated extraction of Tridacna shell growth patterns via machine learning for enhanced paleoclimate/paleoweather research

Tridacna spp. are valuable archives for paleoclimate and paleoweather research due to their distinct daily growth patterns and the sensitivity of the daily growth patterns to environment changes. However, manually identifying daily growth lines and measuring the daily growth increment width (DGIW) of Tridacna shells from Laser Scanning Confocal Microscopy (LSCM) images is a tedious task that has become a significant barrier to Tridacna studies. This paper addresses this challenge by integrating machine learning into Tridacna research for the first time to automate the calculation of the number of daily growth lines and DGIW of Tridacna shells. Specifically, we propose an unsupervised generative adversarial attention network called TriGAN to automatically recognize distinct daily growth lines of Tridacna shells from LSCM images. Utilizing modern Tridacna specimens collected from the South China Sea, our experimental results demonstrate that TriGAN can effectively reconstruct the ambiguous and blurred regions in LSCM images and produce higher quality images of daily growth patterns compared to existing image generation networks. Furthermore, the daily growth line number and DGIW of Tridacna shells can be counted automatically from the images recognized by TriGAN, which are in good agreement with the statistical results obtained manually from the original LSCM images (R = 0.7, p < 0.01 for the DGIW profile of T. gigas specimen MD1 and R = 0.6, p < 0.01 for T. derasa specimen XB10). This automated method provides an efficient solution for researching the laminar chronology of Tridacna.

Stress Detection with Natural Language Processing Techniques from Social Media Articles

Abstract— Stress have significant effects on health and well-being. Therefore, NLP's ability to identify stress situations is critical to providing early intervention and support. NLP can extract meaningful clues from people's online communications and written texts; This makes it a powerful tool for detecting signs of stress. Identifying signs of stress early can help individuals improve their ability to cope with stress by providing appropriate support and guidance; NLP can automate this process. Considering these situations, this study tried to detect the stress states of the users using the Stress Detection from Social Media Articles dataset. Generative Pre-trained Transformer (GPT), Logistic Regression (LR) and Artificial Neural Network (ANN) methods were used for classification. Doc2vec method was used to convert texts into vector data. In the classification processes, the highest classification success was obtained from the ANN model with 80.34%. It is thought that this success can be increased with more data and different models.

Efficient Generation of Conformer Ensembles Using Internal Coordinates and a Generative Directional Graph Convolution Neural Network.

We present a neural-network-based high-throughput molecular conformer-generation algorithm. A chemical graph-convolutional network is trained to predict low-energy conformers in internal coordinate representation (bond lengths, bond, and torsion angles), starting from two-dimensional (2D) chemical topology. Generative neural network (NN) architecture performs denoising from torsion space, producing conformer ensembles with populations that are well correlated with torsion energy profiles. Short force-field-based energy minimization is applied to refine final conformers. All computation-intensive stages of the algorithm are GPU-optimized. The procedure (termed GINGER) is benchmarked on a commonly used test set of bioactive three-dimensional (3D) conformers from the PDB. We demonstrate highly competitive results in conformer recovery and throughput rates suitable for giga-scale compound library processing. A web server that allows interactive conformer ensemble generation by GINGER and their viewing is made freely available at https://www.molsoft.com/gingerdemo.html.

Application of SOM-GAN based deep learning technology in the security protection of rural bank depositors' funds information

Since the reform and opening up, the status of China's banking industry has been significantly improved, and the traditional banking industry has gradually cross-integrated with Internet technology [1], which has become an important support for national economic development and a core component of the financial system. For example, in the original physical bank and online bank, new Internet financial models have been expanded, such as online financial management, bank electronic accounts, mobile banking, P2P models, third-party payment platforms, crowdfunding, etc. [2]. However, in the period of rapid development of the transformation and upgrading of traditional banks facilitated by information technology, many problems have emerged, such as the relaxation of supervision, the lag of policies and regulations, and the difficulty of reasonable protection of private data, etc. These problems may affect the interests of depositors and even the state. For example, in 2022, China's Henan rural bank 40 billion event, this event involved about 400,000 depositors, and platforms such as Du Xiaoman and Xiaomi Finance have related cooperation with these rural banks. Therefore, reasonable security protection of depositors' fund information and information processing and encryption under certain deep learning technology have long become the focus of experts' research in information technology. For example, Tian Gengwen has conducted an in-depth discussion on customer information security in the digital transformation of banks [3]. For example, Yang Wanrong conducted an in-depth study on the path and measures of data security governance of small and medium-sized banks [4]. For example, Zhang Wenli, Peng Xiaolei et al have made an in-depth elaboration on the bank data security solution [5]. Following hot issues closely, this paper provides a method of deep learning technology based on SOM-GAN in the security protection of rural bank depositors' funds information, aiming to combine the advantages of self-organizing mapping neural network and generative adantagonistic network unsupervised learning. At the same time, it can complement the competition mode of self-organized neural network, generate the discriminant characteristics of adversarial network and generate real data to process and review the relevant data of the fund information security of rural bank depositors, ensure the fund management security of rural bank depositors, and realize the monitoring and analysis of their funds and the surrounding environment when depositors are storing funds. Thus, the security, confidentiality, authenticity and integrity of depositors' funds information are guaranteed, and hidden dangers such as theft of depositors' funds, loss of depositors' personal information and threats to depositors' personal information security brought about by the 40bn deposit red code incident of Henan Bank in 2022 are eliminated.

Deep convolutional generative adversarial network for generation of computed tomography images of discontinuously carbon fiber reinforced polymer microstructures

Computed tomography images are of utmost importance when characterizing the heterogeneous and complex microstructure of discontinuously fiber reinforced polymers. However, the devices are expensive and the scans are time- and energy-intensive. Through recent advances in generative adversarial networks, the instantaneous generation of endless numbers of images that are representative of the input images and hold physical significance becomes possible. Hence, this work presents a deep convolutional generative adversarial network trained on approximately 30,000 input images from carbon fiber reinforced polyamide 6 computed tomography scans. The challenge lies in the low contrast between the two constituents caused by the close proximity of the density of polyamide 6 and carbon fibers as well as the small fiber diameter compared to the necessary resolution of the images. In addition, the stochastic, heterogeneous microstructure does not follow any logical or predictable rules exacerbating their generation. The quality of the images generated by the trained network of 256 pixel ×\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\ imes$$\\end{document} 256 pixel was investigated through the Fréchet inception distance and nearest neighbor considerations based on Euclidean distance and structural similarity index measure. Additional visual qualitative assessment ensured the realistic depiction of the complex mixed single fiber and fiber bundle structure alongside flow-related physically feasible positioning of the fibers in the polymer. The authors foresee additionally huge potential in creating three-dimensional representative volume elements typically used in composites homogenization.

W-DRAG: A joint framework of WGAN with data random augmentation optimized for generative networks for bone marrow edema detection in dual energy CT

Dual-energy computed tomography (CT) is an excellent substitute for identifying bone marrow edema in magnetic resonance imaging. However, it is rarely used in practice owing to its low contrast. To overcome this problem, we constructed a framework based on deep learning techniques to screen for diseases using axial bone images and to identify the local positions of bone lesions. To address the limited availability of labeled samples, we developed a new generative adversarial network (GAN) that extends expressions beyond conventional augmentation (CA) methods based on geometric transformations. We theoretically and experimentally determined that combining the concepts of data augmentation optimized for GAN training (DAG) and Wasserstein GAN yields a considerably stable generation of synthetic images and effectively aligns their distribution with that of real images, thereby achieving a high degree of similarity. The classification model was trained using real and synthetic samples. Consequently, the GAN technique used in the diagnostic test had an improved F1 score of approximately 7.8% compared with CA. The final F1 score was 80.24%, and the recall and precision were 84.3% and 88.7%, respectively. The results obtained using the augmented samples outperformed those obtained using pure real samples without augmentation. In addition, we adopted explainable AI techniques that leverage a class activation map (CAM) and principal component analysis to facilitate visual analysis of the network’s results. The framework was designed to suggest an attention map and scattering plot to visually explain the disease predictions of the network.

Deep Multimodal Data Fusion

Multimodal Artificial Intelligence (Multimodal AI), in general, involves various types of data (e.g., images, texts, or data collected from different sensors), feature engineering (e.g., extraction, combination/fusion), and decision-making (e.g., majority vote). As architectures become more and more sophisticated, multimodal neural networks can integrate feature extraction, feature fusion, and decision-making processes into one single model. The boundaries between those processes are increasingly blurred. The conventional multimodal data fusion taxonomy (e.g., early/late fusion), based on which the fusion occurs in, is no longer suitable for the modern deep learning era. Therefore, based on the main-stream techniques used, we propose a new fine-grained taxonomy grouping the state-of-the-art (SOTA) models into five classes: Encoder-Decoder methods, Attention Mechanism methods, Graph Neural Network methods, Generative Neural Network methods, and other Constraint-based methods. Most existing surveys on multimodal data fusion are only focused on one specific task with a combination of two specific modalities. Unlike those, this survey covers a broader combination of modalities, including Vision + Language (e.g., videos, texts), Vision + Sensors (e.g., images, LiDAR), and so on, and their corresponding tasks (e.g., video captioning, object detection). Moreover, a comparison among these methods is provided, as well as challenges and future directions in this area.

PCSAGAN: a physics-constrained generative network based on self-attention for high-fidelity flow field reconstruction

PCSAGAN: a physics-constrained generative network based on self-attention for high-fidelity flow field reconstruction

ВОЗМОЖНОСТИ ИСПОЛЬЗОВАНИЯ ГРАФОВЫХ НЕЙРОННЫХ СЕТЕЙ КАК ИНТЕЛЛЕКТУАЛЬНЫХ ПОМОЩНИКОВ ПРИ ПОСТРОЕНИИ ДИНАМИЧЕСКИХ КОГНИТИВНЫХ КАРТ

Approaches to the study of complex dynamic systems are considered. Methods of graph construc-tion with the help of neural networks are considered. The possibility of using trained generative neural networks to help in building dynamic cognitive maps is presented

Load forecasting based on dynamic adaptive and adversarial graph convolutional networks

With the widespread integration of renewable energy sources into the power system and the diversification and complexity of the energy consumption patterns, which makes it more difficult to maintain a source-load balance of the power system.In the power systems, the results of the power load forecasting can be used for generator scheduling and load distribution. Therefore, the accurate power load forecasting is critical to operation of the power system. In this paper, a load forecasting model based on a dynamic adaptive and adversarial graph convolutional network (DAAGCN) is proposed, which combines a dynamic adaptive graph generation network (DAGG) and a generative adversarial network (GAN). Firstly, DAGG utilizes integrated time-varying embedding and node embedding to generate dynamic adaptive graphs for inferring spatial–temporal dependencies between different temporal loads. Second, the underlying spatial–temporal prediction model is adversarial trained based on the idea of a zero-sum game. The module parameters are optimized using the L1 loss and the adversarial training loss together as the training objectives of the model. Finally, short-term forecasting of electricity loads was performed on the datasets using DAAGCN, and the results were compared with several other forecasting modules.

Score-based diffusion models for generating liquid argon time projection chamber images

For the first time, we show high-fidelity generation of Liquid Argon Time Projection Chamber (LArTPC-like) data using a generative neural network. This demonstrates that methods developed for natural images do transfer to LArTPC-produced images, which, in contrast to natural images, are globally sparse but locally dense. We present the score-based diffusion method employed. We evaluate the fidelity of the generated images using several quality metrics, including modified measures used to evaluate natural images, comparisons between high-dimensional distributions, and comparisons relevant to LArTPC experiments. Published by the American Physical Society 2024

Optimization of a unileg thermoelectric generator by the combination of Taguchi method and evolutionary neural network for green power generation

The performance optimization of thermoelectric generators (TEGs) is crucial for efficiently converting waste heat to green energy. This study analyzes the performance and thermal stress of a unileg thermoelectric module by combining the Taguchi method, analysis of variance (ANOVA), and artificial neural network. The results of the Taguchi method indicate that the leg geometry exerts a negligible effect on output power (OP) and conversion efficiency but significantly affects the thermal stress (TS). The analysis of the objective function (maximum OP-to-maximum TS ratio, denoted by P/S) reveals the optimal configuration results in a P/S value of 0.01255 W‧MPa−1, accompanied by a maximum output power (2.9 W) and thermal stress (232.32 MPa). The maximum thermal stress of the rectangle TE leg (182.29 MPa) is less than that of the triangle (279.85 MPa) by 34.86 %. The Taguchi method, analysis of variance, and artificial neural network have the same sensitivity order on P/S with leg height > leg geometry > copper thickness > ceramic thickness. Furthermore, an evolutionary neural network (ENN) method with five-step optimization from 4000 datasets has been developed to improve machine learning prediction. The average error of P/S value and conversion efficiency from ENN is 1.61 % and 0.35 %, respectively. However, for the ENN predictions, the relative errors of P/S value and efficiency between predictions and simulations are 1.28 % and 0.19 %, respectively. In conclusion, this work has demonstrated the numerical optimization of the performance of unileg TEM for the first time. The use of ENN can predict the performance of TEM precisely, saving running time and markedly reducing computational costs. Moreover, the results are conducive to designing high-performance TEGs with long lifespans.

DuaPIN: Auxiliary task enhanced dual path interaction network for civil court view generation

Civil court view generation (CCVG) is a novel but important task for legal intelligence that aims to automatically generate a judge’s opinion based on the plaintiff’s claims and fact descriptions to interpret the judgment result. The task is more challenging than criminal court view generation as the latter generates views based only on criminal facts as input, whereas the CCVG must consider both the plaintiff’s claims and civil facts under the principle of “no claim, no trial.” However, current approaches still follow criminal domain practices to solve problems in civil cases. Moreover, the explicit modeling of the potential correspondence between claims and facts has often been neglected, as court views are required to respond to each corresponding claim based on factual evidence. To address the issues, we propose a dual path interaction network augmented by two self-supervised auxiliary tasks (named DuaPIN), which follows a bionic design by simulating the thinking logic of judges when writing opinions. Specifically, we construct a structurally symmetric Transformer-based dual path multi-encoder–decoder model such that the two inputs, claim and fact, contribute equally to the generation of civil court views. Moreover, an auxiliary task enhanced (ATE) training paradigm using multiple DuaPIN decoders is proposed to explicitly model the potential correspondence between claims and facts. Extensive experiments on public legal document dataset demonstrated that DuaPIN achieves competitive performance compared with previous methods and offers certain performance improvements to popular pre-trained language models via the ATE training method.

Generative adversarial networks for anonymous acneic face dataset generation.

It is well known that the performance of any classification model is effective if the dataset used for the training process and the test process satisfy some specific requirements. In other words, the more the dataset size is large, balanced, and representative, the more one can trust the proposed model's effectiveness and, consequently, the obtained results. Unfortunately, large-size anonymous datasets are generally not publicly available in biomedical applications, especially those dealing with pathological human face images. This concern makes using deep-learning-based approaches challenging to deploy and difficult to reproduce or verify some published results. In this paper, we propose an efficient method to generate a realistic anonymous synthetic dataset of human faces, focusing on attributes related to acne disorders at three distinct levels of severity (Mild, Moderate, and Severe). Notably, our approach initiates from a small dataset of facial acne images, leveraging generative techniques to augment and diversify the dataset, ensuring comprehensive coverage of acne severity levels while maintaining anonymity and realism in the synthetic data. Therefore, a specific hierarchy StyleGAN-based algorithm trained at distinct levels is considered. Moreover, the utilization of generative adversarial networks for augmentation offers a means to circumvent potential privacy or legal concerns associated with acquiring medical datasets. This is attributed to the synthetic nature of the generated data, where no actual subjects are present, thereby ensuring compliance with privacy regulations and legal considerations. To evaluate the performance of the proposed scheme, we consider a CNN-based classification system, trained using the generated synthetic acneic face images and tested using authentic face images. Consequently, we show that an accuracy of 97.6% is achieved using InceptionResNetv2. As a result, this work allows the scientific community to employ the generated synthetic dataset for any data processing application without restrictions on legal or ethical concerns. Moreover, this approach can also be extended to other applications requiring the generation of synthetic medical images.