Deep Generative Neural Networks Research Articles

Green and blue infrastructure as model system for emissions of technology-critical elements

Over the recent decades, technological advancements have led to a rise in the use of so-called technology-critical elements (TCEs). Environmental monitoring of TCEs forms the base to assess whether this leads to increased anthropogenic release and to public health implications. This study employs an exploratory approach to investigate the distribution of the TCEs Li, Be, V, Ga, Ge, Nb, Sb, Te, Ta, Tl, Bi and the REYs (rare-earth elements including yttrium) in urban aerosol in the city of Vienna, Austria.Leaf samples (n = 292) from 8 plant species and two green facades and water samples (n = 18) from the Wienfluss river were examined using inductively coupled plasma tandem mass spectrometry (ICP-MS/MS). Surface dust contributions were assessed by washing one replicate of each leaf sample and analysing the washing water (n = 146). The impacts of sampling month, plant species and storey level on elemental distribution were assessed by statistical tools and generative deep neural network modelling. Higher TCE levels, including Li, V, Ga, Ge, Tl, Bi, and the REYs, were found in the winter months, likely due to the use of de-icing materials and fossil fuel combustion. A. millefolium and S. heufleriana displayed the highest levels of Li and Ge, respectively. In addition, increased elemental accumulation at lower storeys was observed, including Be, Sb, Bi and the REYs, indicating greater atmospheric dust deposition and recirculation closer to ground level.The results suggest a broad association of TCE levels with urban dust. This study enhances the current understanding of TCE distribution in urban settings and underscores the importance of their inclusion in pollution monitoring. It highlights the complex interplay of human activities, urban infrastructure, and environmental factors, offering valuable insights for managing urban environmental health risks and underlining the need for comprehensive urban ecosystem studies.

Bayesian DivideMix++ for enhanced learning with noisy labels

Leveraging inexpensive and human intervention-based annotating methodologies, such as crowdsourcing and web crawling, often leads to datasets with noisy labels. Noisy labels can have a detrimental impact on the performance and generalization of deep neural networks. Robust models that are able to handle and mitigate the effect of these noisy labels are thus essential. In this work, we explore the open challenges of neural network memorization and uncertainty in creating robust learning algorithms with noisy labels. To overcome them, we propose a novel framework called “Bayesian DivideMix++” with two critical components: (i) DivideMix++, to enhance the robustness against memorization and (ii) Monte-Carlo MixMatch, which focuses on improving the effectiveness towards label uncertainty. DivideMix++ improves the pipeline by integrating the warm-up and augmentation pipeline with self-supervised pre-training and dedicated different data augmentations for loss analysis and backpropagation. Monte-Carlo MixMatch leverages uncertainty measurements to mitigate the influence of uncertain samples by reducing their weight in the data augmentation MixMatch step. We validate our proposed pipeline using four datasets encompassing various synthetic and real-world noise settings. We demonstrate the effectiveness and merits of our proposed pipeline using extensive experiments. Bayesian DivideMix++ outperforms the state-of-the-art models by considerable differences in all experiments. Our findings underscore the potential of leveraging these modifications to enhance the performance and generalization of deep neural networks in practical scenarios.

Audio–video syncing with lip movements using generative deep neural networks

Audio–video syncing with lip movements using generative deep neural networks

Generalize Deep Neural Networks With Adaptive Regularization for Classifying

Regularization is a crucial technology to improve the generalization of deep neural networks. However, traditional regularization method approaches are all scenario-specific, because they are generally with ingeniously designed feature representations from input layer, hidden layer, and output layer, which increase the difficulty of model development and interpretation. To this end, a novel practical and flexible regularization method is presented to obtain higher generalization and interpretability. Specifically, the feature maps are decoupled by global suppression and partial suppression from various scales and locate the salient feature with strong low-resolution semantic information. Moreover, the guided discarding specification for feature decoupling by measuring the feature contributions to network decisions, leads to the logics with better interpretability. Subsequently, the max values of the feature map are suppressed by discarding the corresponding salient features. Comprehensive experiments demonstrate that the proposed adaptive regularization outperforms the state-of-the-art performance in image classification accuracy, generalization, and interpretability on several widely used datasets. And adaptive regularization helps the network to mine the connection between salient features, nonsalient features, and ground truth, encouraging the network to construct multiple layers of feature associations.

Deep unfolding as iterative regularization for imaging inverse problems

Deep unfolding methods have gained significant popularity in the field of inverse problems as they have driven the design of deep neural networks (DNNs) using iterative algorithms. In contrast to general DNNs, unfolding methods offer improved interpretability and performance. However, their theoretical stability or regularity in solving inverse problems remains subject to certain limitations. To address this, we reevaluate unfolded DNNs and observe that their algorithmically-driven cascading structure exhibits a closer resemblance to iterative regularization. Recognizing this, we propose a modified training approach and configure termination criteria for unfolded DNNs, thereby establishing the unfolding method as an iterative regularization technique. Specifically, our method involves the joint learning of a convex penalty function using an input-convex neural network to quantify distance to a real data manifold. Then, we train a DNN unfolded from the proximal gradient descent algorithm, incorporating this learned penalty. Additionally, we introduce a new termination criterion for the unfolded DNN. Under the assumption that the real data manifold intersects the solutions of the inverse problem with a unique real solution, even when measurements contain perturbations, we provide a theoretical proof of the stable convergence of the unfolded DNN to this solution. Furthermore, we demonstrate with an example of magnetic resonance imaging reconstruction that the proposed method outperforms original unfolding methods and traditional regularization methods in terms of reconstruction quality, stability, and convergence speed.

Feature Noise Boosts DNN Generalization Under Label Noise.

The presence of label noise in the training data has a profound impact on the generalization of deep neural networks (DNNs). In this study, we introduce and theoretically demonstrate a simple feature noise (FN) method, which directly adds noise to the features of training data and can enhance the generalization of DNNs under label noise. Specifically, we conduct theoretical analyses to reveal that label noise leads to weakened DNN generalization by loosening the generalization bound, and FN results in better DNN generalization by imposing an upper bound on the mutual information between the model weights and the features, which constrains the generalization bound. Furthermore, we conduct a qualitative analysis to discuss the ideal type of FN that obtains good label noise generalization. Finally, extensive experimental results on several popular datasets demonstrate that the FN method can significantly enhance the label noise generalization of state-of-the-art methods. The source codes of the FN method are available on https://github.com/zlzenglu/FN.

Applying a Method for Augmenting Data Mixed from Two Different Sources Using Deep Generative Neural Networks to Management Science

Although a multimodal data analysis, comprising physiological and questionnaire survey data, provides better insights into addressing management science concerns, such as challenging the predictions of consumer choice behavior, studies in this field are scarce because of two obstacles: limited sample size and information privacy. This study addresses these challenges by synthesizing multimodal data using deep generative models. We obtained multimodal data by conducting an electroencephalography (EEG) experiment and a questionnaire survey on the prediction of skilled nurses. Subsequently, we validated the effectiveness of the synthesized data compared with real data regarding the similarities between these data and the predictive performance. We confirmed that the synthesized big data were almost equal to the real data using the trained models through sufficient epochs. Conclusively, we demonstrated that synthesizing data using deep generative models might overcome two significant concerns regarding multimodal data utilization, including physiological data. Our approach can contribute to the prevailing combined big data from different modalities, such as physiological and questionnaire survey data, when solving management issues.

Applications and Evaluation of AI Technologies in the Renovation of Old Buildings in Urban Centers

Urban sprawl or expansion is a growing issue globally, and as technology continues to develop, the demand for residential areas continues to rise. Conflicts regarding the availability of resources for architecture also rise in rural areas; therefore, renovating old buildings in urban centers becomes an important concept. Since there are many challenges with renovating these buildings, the recent development of AI technologies will provide solutions to mitigate or reduce these challenges. This paper examines the application of AI technologies in architecture and how they contribute to redeveloping old buildings in urban areas. Technologies such as conditional Generative Adversarial Networks and deep neural networks can assist architects in standardizing floor plans and generating design modules. In addition, these technologies analyze data from existing buildings and, thus, can create the best possible plan for a building. While these technologies focus on simplifying the traditional process of architectural planning in areas such as generating programs and creating partitions, B-SMART reference architecture approaches architecture from the perspective of smart home building, therefore, implementing AI technologies into the lives of the residents. Furthermore, with AI technologies, future homes are created to be energy efficient and sustainable. Therefore, the application of AI will improve the sustainability of homes and overall benefit the environment.

Visual image reconstruction based on EEG signals using a generative adversarial and deep fuzzy neural network

The decoding of the brain stimulated by visual images has been discussed in this study in order to classify and reconstruct the stimulating images. The dataset utilized was given by Stanford University and included the EEG signal recorded from ten healthy people. In the first stage, the proposed approach combines the ability of the Long-Short-Term Memory (LSTM) network to extract sequence (EEG signals) features with the ability of fuzzy network to extract features in high-uncertainty contexts. The features extracted in the previous stage, which is a noise-free and compressed form of the EEG data, were employed as a conditional vector in the CGAN network in the following step. As a result, this network was used to reconstruct visual stimuli that evoke specific brain responses. When compared to earlier research, the evaluation results produced utilizing the suggested approach demonstrate a greater classification accuracy, precision, recall, and F1 score. Also, the similarity of produced and real images was measured quantitatively using two parameters, FID and SSIM; the SSIM results for 10 participants were on average 0.37, 0.35, 0.35, 0.34, 0.42, and 0.42 for groups (1) to (6), and the FID results were 0.44, 0.45, 0.51, 0.50, 0.39 and 0.39, respectively. Therefore, it can be stated that the method suggested in this study can reconstruct visual stimuli using brain response (EEG signal) with acceptable accuracy, so taking a significant step towards the objective of mind reading.

Towards Better Generalization of Deep Neural Networks via Non-Typicality Sampling Scheme.

Improving the generalization performance of deep neural networks (DNNs) trained by minibatch stochastic gradient descent (SGD) has raised lots of concerns from deep learning practitioners. The standard simple random sampling (SRS) scheme used in minibatch SGD treats all training samples equally in gradient estimation. In this article, we study a new data selection method based on the intrinsic property of the training set to help DNNs have better generalization performance. Our theoretical analysis suggests that this new sampling scheme, called the nontypicality sampling scheme, boosts the generalization performance of DNNs through biasing the solution toward wider minima, under certain assumptions. We confirm our findings experimentally and show that more variants of minibatch SGD can also benefit from the new sampling scheme. Finally, we discuss an extension of the nontypicality sampling scheme that holds promise to enhance both generalization performance and convergence speed of minibatch SGD.

Simulation-aided machine learning: Applications in underwater acoustic perception and explainability

Underwater acoustic applications of machine learning are challenging due to sparsity, lack of diversity, and often uncertain labeling of training data. This is due to low levels of consumer relevance. One approach to the problem is to collect and label more data. However, this is expensive and unlikely to fully address the issue. A complementary approach is to use simulation for generating synthetic data and augmenting the machine learning pipeline. This also enables explainability by encoding human knowledge of the underlying physics. We present two case studies in simulation-aided machine learning. The first is an application to underwater munition dumpsite characterization using multi-view sonar surveys. Self-supervised learning has been used to pre-train a generative deep neural network to predict alternate views using unlabeled data. Subsequently, training is completed using a small volume of labelled data. Simulation was necessary for successful pre-training to continuously fill viewing angle gaps caused by limited and rigid survey directions. The second application is autonomous tactical planning for uncrewed underwater vehicles. Reinforcement learning policies have been trained to passively detect and localize underwater acoustic sources. These have been trained and evaluated in a fully synthetic environment, using a sophisticated propagation model together with empirical environmental parameters.

Normalization Techniques in Training DNNs: Methodology, Analysis and Application.

Normalization techniques are essential for accelerating the training and improving the generalization of deep neural networks (DNNs), and have successfully been used in various applications. This paper reviews and comments on the past, present and future of normalization methods in the context of DNN training. We provide a unified picture of the main motivation behind different approaches from the perspective of optimization, and present a taxonomy for understanding the similarities and differences between them. Specifically, we decompose the pipeline of the most representative normalizing activation methods into three components: the normalization area partitioning, normalization operation and normalization representation recovery. In doing so, we provide insight for designing new normalization technique. Finally, we discuss the current progress in understanding normalization methods, and provide a comprehensive review of the applications of normalization for particular tasks, in which it can effectively solve the key issues.

Delineation of Integrated Anomaly with Generative Adversarial Networks and Deep Neural Networks in the Zhaojikou Pb‐Zn Ore District, Southeast China

AbstractGeochemical maps are of great value in mineral exploration. Integrated geochemical anomaly maps provide comprehensive information about mapping assemblages of element concentrations to possible types of mineralization/ore, but vary depending on expert's knowledge and experience. This paper aims to test the capability of deep neural networks to delineate integrated anomaly based on a case study of the Zhaojikou Pb‐Zn deposit, Southeast China. Three hundred fifty two samples were collected, and each sample consisted of 26 variables covering elemental composition, geological, and tectonic information. At first, generative adversarial networks were adopted for data augmentation. Then, DNN was trained on sets of synthetic and real data to identify an integrated anomaly. Finally, the results of DNN analyses were visualized in probability maps and compared with traditional anomaly maps to check its performance. Results showed that the average accuracy of the validation set was 94.76%. The probability maps showed that newly‐identified integrated anomalous areas had a probability of above 75% in the northeast zones. It also showed that DNN models that used big data not only successfully recognized the anomalous areas identified on traditional geochemical element maps, but also discovered new anomalous areas, not picked up by the elemental anomaly maps previously.

Steganography of Steganographic Networks

Steganography is a technique for covert communication between two parties. With the rapid development of deep neural networks (DNN), more and more steganographic networks are proposed recently, which are shown to be promising to achieve good performance. Unlike the traditional handcrafted steganographic tools, a steganographic network is relatively large in size. It raises concerns on how to covertly transmit the steganographic network in public channels, which is a crucial stage in the pipeline of steganography in real world applications. To address such an issue, we propose a novel scheme for steganography of steganographic networks in this paper. Unlike the existing steganographic schemes which focus on the subtle modification of the cover data to accommodate the secrets. We propose to disguise a steganographic network (termed as the secret DNN model) into a stego DNN model which performs an ordinary machine learning task (termed as the stego task). During the model disguising, we select and tune a subset of filters in the secret DNN model to preserve its function on the secret task, where the remaining filters are reactivated according to a partial optimization strategy to disguise the whole secret DNN model into a stego DNN model. The secret DNN model can be recovered from the stego DNN model when needed. Various experiments have been conducted to demonstrate the advantage of our proposed method for covert communication of steganographic networks as well as general DNN models.

Generative Adversarial Networks for Cyber Threat Hunting in Ethereum Blockchain

Ethereum blockchain has shown great potential in providing the next generation of the decentralized platform beyond crypto payments. Recently, it has attracted researchers and industry players to experiment with developing various Web3 applications for the Internet of Things (IoT), Defi, Metaverse, and many more. Although Ethereum provides a secure platform for developing decentralized applications, it is not immune to security risks and has been a victim of numerous cyber attacks. Adversarial attacks are a new cyber threat to systems that have been rising. Adversarial attacks can disrupt and exploit decentralized applications running on the Ethereum platform by creating fake accounts and transactions. Detecting adversarial attacks is challenging because the fake materials (e.g., accounts and transactions) as malicious payloads are similar to benign data. This article proposes a model using Generative Adversarial Networks (GAN) and Deep Recurrent Neural Networks (RNN) for cyber threat hunting in the Ethereum blockchain. Firstly, we employ GAN to generate fake transactions using genuine Ethereum transactions as the first phase of the proposed model. Then in the second phase, we utilize bi-directional Long Short-Term Memory (LSTM) to identify adversarial transactions in a hunting exercise. The results of the first phase evaluation show that the GAN can generate transactions identical to the actual Ethereum transactions with an accuracy of 82.51%. Also, the results of the second phase show 99.98% accuracy in identifying adversarial transactions.

ScSemiAAE: a semi-supervised clustering model for single-cell RNA-seq data

BackgroundSingle-cell RNA sequencing (scRNA-seq) strives to capture cellular diversity with higher resolution than bulk RNA sequencing. Clustering analysis is critical to transcriptome research as it allows for further identification and discovery of new cell types. Unsupervised clustering cannot integrate prior knowledge where relevant information is widely available. Purely unsupervised clustering algorithms may not yield biologically interpretable clusters when confronted with the high dimensionality of scRNA-seq data and frequent dropout events, which makes identification of cell types more challenging.ResultsWe propose scSemiAAE, a semi-supervised clustering model for scRNA sequence analysis using deep generative neural networks. Specifically, scSemiAAE carefully designs a ZINB adversarial autoencoder-based architecture that inherently integrates adversarial training and semi-supervised modules in the latent space. In a series of experiments on scRNA-seq datasets spanning thousands to tens of thousands of cells, scSemiAAE can significantly improve clustering performance compared to dozens of unsupervised and semi-supervised algorithms, promoting clustering and interpretability of downstream analyses.ConclusionscSemiAAE is a Python-based algorithm implemented on the VSCode platform that provides efficient visualization, clustering, and cell type assignment for scRNA-seq data. The tool is available from https://github.com/WHang98/scSemiAAE.

Explainable semi-supervised deep learning shows that dementia is associated with small, avocado-shaped clocks with irregularly placed hands

The clock drawing test is a simple and inexpensive method to screen for cognitive frailties, including dementia. In this study, we used the relevance factor variational autoencoder (RF-VAE), a deep generative neural network, to represent digitized clock drawings from multiple institutions using an optimal number of disentangled latent factors. The model identified unique constructional features of clock drawings in a completely unsupervised manner. These factors were examined by domain experts to be novel and not extensively examined in prior research. The features were informative, as they distinguished dementia from non-dementia patients with an area under receiver operating characteristic (AUC) of 0.86 singly, and 0.96 when combined with participants’ demographics. The correlation network of the features depicted the “typical dementia clock” as having a small size, a non-circular or “avocado-like” shape, and incorrectly placed hands. In summary, we report a RF-VAE network whose latent space encoded novel constructional features of clocks that classify dementia from non-dementia patients with high performance.

Extreme and severe convective weather disasters: A dual-polarization radar nowcasting method based on physical constraints and a deep neural network model

It is difficult for traditional methods, such as the PySTEPS method, which is based on optical flows, and the UNet method, which is based on convolutional neural networks, to predict extreme weather events (e.g., hail, heavy rain, lightning and mesoscale cyclones) 0–60 min in advance because these events have short life cycles, occur at small scales, rapidly develop and are highly nonlinear. Based on the structural characteristics of severe weather, we propose a physically constrained generative adversarial deep neural network nowcasting model to improve the prediction accuracy and blurry data problem associated with nowcasting extreme convective weather within the 0–60 min interval. The improved TITAN algorithm is used to quickly identify and match strong storm cells in three dimensions, and the convection-diffusion constraint equation is established. Additionally, reflectivity (Zh), differential reflectivity (Zdr), the range derivative-specific differential phase (Kdp), correlation coefficient (cc), precipitation particle phase, and velocity divergence are considered when training the model with a large sample set, and constraint equations are used for optimization. The critical success index, false alarm ratio and miss probability scores of the nowcasting results obtained with this method are better than the PySTEPS and UNet results. Additionally, the root mean square error of a hailstorm nowcasted with a 30-min lead time is significantly better than that of the other methods, and the blurry data problem that appears over time is mitigated. We show that our model can provide nowcasts with high precision to support operations at various resolutions and lead times in cases for which alternative methods struggle.

Helium focused ion beam induced subsurface damage on Si and SiC substrates: experiments and generative deep neural network modeling via position-dependent input

Deep neural networks (DNNs) are reshaping many fields due to their end-to-end ability to learn directly from data, leading to outstanding performance, especially in image processing. However, their training requires large image datasets while image preparation can be expensive/time-consuming. If the underlying behavior can be formulated as spatial-dependent, we propose that every pixel from every image can be considered as a different data source, thus enabling a truly large pixel-based dataset with millions of elements (‘position-dependent input’), even when obtained from a few images. The approach is applied to helium focused ion beam nanofabrication, where the cross-section of the helium-damaged region essentially resembles that of a lightbulb, while close inspection reveals the presence of (i) an outer defective region in direct contact with the bulk substrate and (ii) an inner amorphous region filled with helium bubbles of gradually increasing size. Interestingly, the amorphous phase may swell upwards to form a protruding mesa, depending on the beam energy and dose, a feature that has resisted modeling so far. Through dedicated experiments on both Si and SiC substrates, and careful image analysis (segmentation), we describe the transformations that take place in the defective and amorphous regions with increasing energy and dose, and use the outlined position-dependent input together with a simple DNN of 4 hidden layers and 16 neurons per layer to describe all damage features realistically, inherently demonstrating generative behavior. This is the first time that any model predicts swelling satisfactorily. Generalization is surprisingly smooth and accurate.

Disruption prediction based on fusion feature extractor on J-TEXT

Predicting disruptions across different tokamaks is necessary for next generation device. Future large-scale tokamaks can hardly tolerate disruptions at high performance discharge, which makes it difficult for current data-driven methods to obtain an acceptable result. A machine learning method capable of transferring a disruption prediction model trained on one tokamak to another is required to solve the problem. The key is a feature extractor which is able to extract common disruption precursor traces in tokamak diagnostic data, and can be easily transferred to other tokamaks. Based on the concerns above, this paper presents a deep feature extractor, namely, the fusion feature extractor (FFE), which is designed specifically for extracting disruption precursor features from common diagnostics on tokamaks. Furthermore, an FFE-based disruption predictor on J-TEXT is demonstrated. The feature extractor is aimed to extracting disruption-related precursors and is designed according to the precursors of disruption and their representations in common tokamak diagnostics. Strong inductive bias on tokamak diagnostics data is introduced. The paper presents the evolution of the neural network feature extractor and its comparison against general deep neural networks, as well as a physics-based feature extraction with a traditional machine learning method. Results demonstrate that the FFE may reach a similar effect with physics-guided manual feature extraction, and obtain a better result compared with other deep learning methods.