Semi-supervised Variational Autoencoder Research Articles

A mechanical fault diagnosis model with semi-supervised variational autoencoder based on long short-term memory network

A mechanical fault diagnosis model with semi-supervised variational autoencoder based on long short-term memory network

Spatial and temporal downscaling schemes to reconstruct high-resolution GRACE data: A case study in the Tarim River Basin, Northwest China

Climate change and excessive exploitation of water resources exert pressure on groundwater supply and the ecosystem in drylands. Although The Gravity Recovery and Climate Experiment (GRACE) satellites has demonstrated the feasibility of quantifying global groundwater storage variations, monitoring regional-scale groundwater has been challenging due to the coarse resolution of GRACE. Previous GRACE downscaling studies focused on develop new algorithms based on the perspective of pixel spatial correlation to improve resolution, which cannot better capture the temporal evolution of GRACE data effectively. In this study, we employ the semi-supervised variational autoencoder (SSVAER) algorithm and the multi-scale geographically weighted regression (MGWR) model to establish two different downscaling schemes: pixel temporal continuity downscaling and pixel spatial correlation downscaling. These schemes achieve spatial resolution downscaling of GRACE-derived groundwater storage anomalies (GWSA) from 0.5° to 0.1°. Additionally, the applicability of the PCR-GLOBWB model in drylands is verified. Furtherly, the spatiotemporal distribution patterns of GWSA are analyzed. The results show that (1) Both the temporal and spatial downscaling methods produced consistent results, with data correlations ranged from 0.94 to 0.98 observed in over 80 % of the range before and after downscaling; (2) The groundwater storage change rate in the northern Tarim River Basin (TRB) is 25 times greater than the model results, while in other regions, the average deviation is 2.6 times; (3) The two schemes enhance the correlation (0.27) between GWSA and groundwater level anomaly (GWLA) to 0.59 and 0.52, respectively, with a three-month lag in GWSA relative to GWLA. The temporal downscaling approach exhibited higher CC and lower RMSE, outperforming the spatial downscaling approach. The high-resolution results in this study can well complement groundwater level prediction efforts in arid regions and provide quantitative information for local water resource management.

An evolutionary variational autoencoder for perovskite discovery

Machine learning (ML) techniques emerged as viable means for novel materials discovery and target property determination. At the vanguard of discoverable energy materials are perovskite crystalline materials, which are known for their robust design space and multifunctionality. Previous efforts for simulating the discovery of novel perovskites via ML have often been limited to straightforward tabular-dataset models and compositional phase-field representations. Therefore, the present study makes a contribution in expanding ML capability by demonstrating the efficacy of a new deep evolutionary learning framework for discovering stable and functional inorganic materials that adopts the complex A2BB′X6 and AA′BB′X6 double perovskite stoichiometries. The model design is called the Evolutionary Variational Autoencoder for Perovskite Discovery (EVAPD), which is comprised of a semi-supervised variational autoencoder (SS-VAE), an evolutionary-based genetic algorithm, and a one-to-one similarity analytical model. The genetic algorithm performs adaptive metaheuristic search operations for finding the most theoretically stable candidates emerging from a target-learnable latent space of the generative SS-VAE model. The integrated similarity analytical model assesses the deviation in three-dimensional atomic coordination between newly generated perovskites and proven standards, and as such, recommends the most promising and experimentally feasible candidates. Using Density Functional Theory (DFT), the novel perovskites are subjected to thorough variable-cell optimization and property determination. The current study presents 137 new perovskite materials generated by the proposed EVAPD model and identifies potential candidates for photovoltaic and optoelectronic applications. The new materials data are archived at NOMAD repository (doi.org/10.17172/NOMAD/2023.05.31-1) and are made openly available to interested users.

Robust Semisupervised Deep Generative Model Under Compound Noise.

Semisupervised learning has been widely applied to deep generative model such as variational autoencoder. However, there are still limited work in noise-robust semisupervised deep generative model where the noise exists in both of the data and the labels simultaneously, which are referred to as outliers and noisy labels or compound noise. In this article, we propose a novel noise-robust semisupervised deep generative model by jointly tackling the noisy labels and outliers in a unified robust semisupervised variational autoencoder randomized generative adversarial network (URSVAE-GAN). Typically, we consider the uncertainty of the information of the input data in order to enhance the robustness of the variational encoder toward the noisy data in our unified robust semisupervised variational autoencoder (URSVAE). Subsequently, in order to alleviate the detrimental effects of noisy labels, a denoising layer is integrated naturally into the semisupervised variational autoencoder so that the variational inference is conditioned on the corrected labels. Moreover, to enhance the robustness of the variational inference in the presence of outliers, the robust β-divergence measure is employed to derive the novel variational lower bound, which already achieves competitive performance. This further motivates the development of URSVAE-GAN that collapses the decoder of URSVAE and the generator of a robust semisupervised generative adversarial network into one unit. By applying the end-to-end denoising scheme in the joint optimization, the experimental results demonstrate the superiority of the proposed framework by the evaluating on image classification and face recognition tasks and comparing with the state-of-the-art approaches.

An Improved Semi-supervised Variational Autoencoder with Gate Mechanism for Text Classification

In recent years, semi-supervised learning has been investigated to take full advantages of increasing unlabeled data. Although pretrained deep learning models are successfully adopted on a massive amount of unlabeled data, they may not be applicable in specific domains as the data is limited. In this paper, we propose a model, termed Semi-supervised Variational AutoEncoder (SVAE), which consists of Gated Convolutional Neural Networks (GCNN) as both the encoder and the decoder. Since the canonical VAE suffers from Kullback–Leibler (KL) vanishing problem, we attach a layer named Scalar after Batch Normalization (BN) to scale the output of the BN. We conduct experiments on two domain-specific datasets with a small amount of data. The results show that SVAE outperforms other alternative baselines for language modeling and semi-supervised learning studies. Especially, the results in the language modeling validate the effect of combining BN and Scalar for tackling the KL vanishing problem. Moreover, the visualization of the latent representations verifies the performance of SVAE on less data.

Exploring Low-Toxicity Chemical Space with Deep Learning for Molecular Generation.

Creating a wide range of new compounds that not only have ideal pharmacological properties but also easily pass long-term toxicity evaluation is still a challenging task in current drug discovery. In this study, we developed a conditional generative model by combining a semisupervised variational autoencoder (SSVAE) with an MGA toxicity predictor. Our aim is to generate molecules with low toxicity, good drug-like properties, and structural diversity. For multiobjective optimization, we have developed a method with hierarchical constraints on the toxicity space of small molecules to generate drug-like small molecules, which can also minimize the effect on the diversity of generated results. The evaluation results of the metrics indicate that the developed model has good effectiveness, novelty, and diversity. The generated molecules by this model are mainly distributed in low-toxicity regions, which suggests that our model can efficiently constrain the generation of toxic structures. In contrast to simply filtering toxic ones after generation, the low-toxicity molecular generative model can generate molecules with structural diversity. Our strategy can be used in target-based drug discovery to improve the quality of generated molecules with low-toxicity, drug-like, and highly active properties.

Oxo-degradable plastics worry firms

Quality variables are key indicators of the operating performance in industrial processes. Because they are difficult to measure, soft sensor models can be adopted to predict them timely. For accurate prediction, sufficient training data are necessary to construct a good soft sensor model. In practical industrial processes, however, data labeled with quality variables are usually deficient in the desired region. Particularly, when the process is just switched to a new mode, available data in this new mode are initially quite a few. In this paper, a novel data synthesis method based on the regressor-embedded semi-supervised variational autoencoder (RSSVAE) model is proposed to generate synthetic labeled data when the original labeled data are inadequate. The proposed model utilizes not only the original data in the data-scarce region but also the data in other regions, which share some common information with the scarce data. Meanwhile, data synthesis and model correction mechanism are implemented iteratively to avoid model biases. Once the synthetic labeled data of the data-scarce region are acquired, they are combined with the original labeled data to establish a local soft sensor and predict the quality variables of the unlabeled data. Finally, a real ammonia synthesis process is introduced to demonstrate the effectiveness of the proposed method.

Synthesizing labeled data to enhance soft sensor performance in data-scarce regions

Quality variables are key indicators of the operating performance in industrial processes. Because they are difficult to measure, soft sensor models can be adopted to predict them timely. For accurate prediction, sufficient training data are necessary to construct a good soft sensor model. In practical industrial processes, however, data labeled with quality variables are usually deficient in the desired region. Particularly, when the process is just switched to a new mode, available data in this new mode are initially quite a few. In this paper, a novel data synthesis method based on the regressor-embedded semi-supervised variational autoencoder (RSSVAE) model is proposed to generate synthetic labeled data when the original labeled data are inadequate. The proposed model utilizes not only the original data in the data-scarce region but also the data in other regions, which share some common information with the scarce data. Meanwhile, data synthesis and model correction mechanism are implemented iteratively to avoid model biases. Once the synthetic labeled data of the data-scarce region are acquired, they are combined with the original labeled data to establish a local soft sensor and predict the quality variables of the unlabeled data. Finally, a real ammonia synthesis process is introduced to demonstrate the effectiveness of the proposed method.

A machine learning workflow for molecular analysis: application to melting points

Computational tools encompassing integrated molecular prediction, analysis, and generation are key for molecular design in a variety of critical applications. In this work, we develop a workflow for molecular analysis (MOLAN) that integrates an ensemble of supervised and unsupervised machine learning techniques to analyze molecular data sets. The MOLAN workflow combines molecular featurization, clustering algorithms, uncertainty analysis, low-bias dataset construction, high-performance regression models, graph-based molecular embeddings and attribution, and a semi-supervised variational autoencoder based on the novel SELFIES representation to enable molecular design. We demonstrate the utility of the MOLAN workflow in the context of a challenging multi-molecule property prediction problem: the determination of melting points solely from single molecule structure. This application serves as a case study for how to employ the MOLAN workflow in the context of molecular property prediction.

MixPoet: Diverse Poetry Generation via Learning Controllable Mixed Latent Space

As an essential step towards computer creativity, automatic poetry generation has gained increasing attention these years. Though recent neural models make prominent progress in some criteria of poetry quality, generated poems still suffer from the problem of poor diversity. Related literature researches show that different factors, such as life experience, historical background, etc., would influence composition styles of poets, which considerably contributes to the high diversity of human-authored poetry. Inspired by this, we propose MixPoet, a novel model that absorbs multiple factors to create various styles and promote diversity. Based on a semi-supervised variational autoencoder, our model disentangles the latent space into some subspaces, with each conditioned on one influence factor by adversarial training. In this way, the model learns a controllable latent variable to capture and mix generalized factor-related properties. Different factor mixtures lead to diverse styles and hence further differentiate generated poems from each other. Experiment results on Chinese poetry demonstrate that MixPoet improves both diversity and quality against three state-of-the-art models.

A Semi-Supervised and Incremental Modeling Framework for Wafer Map Classification

Wafer map analysis provides critical information for quality control and yield improvement tasks in semiconductor manufacturing. In particular, wafer patterns of gross failing areas (GFA) are important clues to identify the causes of relevant failures during the manufacturing process. In this work, a semi-supervised classification framework is proposed for wafer map analysis, and its application to wafer bin maps with GFA patterns classification is demonstrated. The Ladder network and the semi-supervised variational autoencoder are adopted to classify wafer bin maps in comparison with a standard convolutional neural network (CNN) model on two real-world datasets. The results have illustrated that two semi-supervised models are consistently and substantially better than the CNN model across various training data percentages by effective utilization of the unlabeled data. Active learning and pseudo labeling are also utilized to accelerate the learning curve.

Zero-shot learning for action recognition using synthesized features

The major disadvantage of supervised methods for action recognition is the need for a large amount of annotated data, where the data is matched to its label accurately. To address this issue, Zero-Shot Learning (ZSL) is introduced. Zero short learning primarily uses data that is synthesized to compensate for lack of training examples. In this paper, two different approaches are proposed for the synthesis of artificial examples for novel classes; namely, inverse autoregressive flow (IAF) based generative model and bi-directional adversarial GAN(Bi-dir GAN). A consequence of the proposed approach is a transductive setting using a semi-supervised variational autoencoder, where the unlabelled data from unseen classes are used to train the model. This enables the generation of novel class examples from textual descriptions. The proposed models perform well in the following settings, namely, i) Standard setting(ZSL), where the test data comes only from unseen classes, and ii) Generalized setting(GZSL), where the test data comes from both seen and unseen classes. In the case of the generalized setting, examples with pseudo labels are generated for unseen classes. Experiments are performed on three baseline datasets, UCF101, HMDB51, and Olympic. In comparison with state-of-the-art approaches, both the proposed models, IAF based generative model and Bi-dir GAN model outperform in UCF101, and Olympic datasets in all the settings and achieve comparative results in HMDB51.

Conditional Molecular Design with Deep Generative Models.

Although machine learning has been successfully used to propose novel molecules that satisfy desired properties, it is still challenging to explore a large chemical space efficiently. In this paper, we present a conditional molecular design method that facilitates generating new molecules with desired properties. The proposed model, which simultaneously performs both property prediction and molecule generation, is built as a semisupervised variational autoencoder trained on a set of existing molecules with only a partial annotation. We generate new molecules with desired properties by sampling from the generative distribution estimated by the model. We demonstrate the effectiveness of the proposed model by evaluating it on drug-like molecules. The model improves the performance of property prediction by exploiting unlabeled molecules and efficiently generates novel molecules fulfilling various target conditions.

Inferring Emotion from Conversational Voice Data: A Semi-Supervised Multi-Path Generative Neural Network Approach

To give a more humanized response in Voice Dialogue Applications (VDAs), inferring emotion states from users’ queries may play an important role. However, in VDAs, we have tremendous amount of VDA users and massive scale of unlabeled data with high dimension features from multimodal information, which challenge the traditional speech emotion recognition methods. In this paper, to better infer emotion from conversational voice data, we proposed a semi-supervised multi-path generative neural network. Specifically, first, we build a novel supervised multi-path deep neural network framework. To avoid high dimensional input, raw features are trained by groups in local classifiers. Then high-level features of each local classifiers are concatenated as input of a global classifier. These two kinds classifiers are trained simultaneously through a single objective function to achieve a more effective and discriminative emotion inferring. To further solve the labeled-data-scarcity problem, we extend the multi-path deep neural network to a generative model based on semi-supervised variational autoencoder (semi-VAE), which is able to train the labeled and unlabeled data simultaneously. Experiment based on a 24,000 real-world dataset collected from Sogou Voice Assistant (SVAD13) and a benchmark dataset IEMOCAP show that our method significantly outperforms the existing state-of-the-art results.

Variational Autoencoder for Semi-Supervised Text Classification

Although semi-supervised variational autoencoder (SemiVAE) works in image classification task, it fails in text classification task if using vanilla LSTM as its decoder. From a perspective of reinforcement learning, it is verified that the decoder's capability to distinguish between different categorical labels is essential. Therefore, Semi-supervised Sequential Variational Autoencoder (SSVAE) is proposed, which increases the capability by feeding label into its decoder RNN at each time-step. Two specific decoder structures are investigated and both of them are verified to be effective. Besides, in order to reduce the computational complexity in training, a novel optimization method is proposed, which estimates the gradient of the unlabeled objective function by sampling, along with two variance reduction techniques. Experimental results on Large Movie Review Dataset (IMDB) and AG's News corpus show that the proposed approach significantly improves the classification accuracy compared with pure-supervised classifiers, and achieves competitive performance against previous advanced methods. State-of-the-art results can be obtained by integrating other pretraining-based methods.