Data Manifold Research Articles

An Attention-Based Cycle-Consistent Generative Adversarial Network for IoT Data Generation and Its Application in Smart Energy Systems

The availability of Internet of Things (IoT) data is essential for the operation of intelligent systems, such as smart energy systems. Unfortunately, information sensitivity and the lack of observations tend to impact the availability of IoT data. To solve this problem, this article proposes an attention-based cycle-consistent generative adversarial network (ABC-GAN) to generate IoT data. By efficiently learning the distribution among different data patterns and sufficiently capturing temporal features, ABC-GAN can effectively reproduce the IoT data collected from different devices and regions. Various experimental results in smart energy systems demonstrate that ABC-GAN excels at capturing the temporal features, distribution, and latent manifolds of the original data when compared to the baselines and that the prediction models trained with the data generated by ABC-GAN can achieve performances similar to models trained with the real data.

Efficient generation of valid test inputs for deep neural networks via gradient search

AbstractThe safety and robustness of deep neural networks (DNNs) are currently of great concern. Adequate testing is commonly an effective technique to ensure the software's trustworthiness. However, existing DNN testing methods generate many invalid test inputs, which inevitably brings increased computational overhead and reduces the efficiency of DNN testing. In this paper, we focus on testing task‐specific DNN and investigating diverse, valid and natural test input generation based on data augmentation techniques. Specifically, we propose AugTest, a DNN testing method based on stochastic optimization with momentum, searching for optimal compositions of data augmentation parameters to efficiently generate diverse and valid test inputs. Experimental results show that our proposed method can effectively explore the data manifold space and find valid test inputs with high diversity and naturalness. Compared with the best‐performing baseline, AugTest can generate more test inputs with more average diversity and less average time. Furthermore, the generated test inputs have competitive generalizability to DNNs with different structures. The test error rates exceed 70% when testing other DNN models performing similar tasks using the test inputs generated by AugTest. This implies that our method can produce more valid and generalized data to unveil DNNs' errors.

Large Data Limit of the MBO Scheme for Data Clustering

AbstractThe MBO scheme is a highly performant scheme used for data clustering and other applications in machine learning. We report on the first theoretical studies of the scheme in the large data limit. Our results relate (i) the final state of the scheme to minimal surfaces in the data manifold and (ii) the dynamics of the scheme to the steepest descent for surfaces, which is mean curvature flow. The tools employed are variational methods for (i) and viscosity solution techniques for (ii).

Fault Diagnosis for Multilevel Converters Based on an Affine-Invariant Riemannian Metric Autoencoder

Multilevel converters play an important role in power electronic systems. In recent years, data-driven fault diagnosis technologies for multilevel converters have rapidly developed and are generally based on feature engineering and intelligent classification algorithms. However, the fault detection rate, speed, and stability still need to be improved, especially for complex converter systems. In this article, based on a probabilistic autoencoder architecture, an affine-invariant Riemannian metric autoencoder (AIRMAE) is proposed for feature extraction, and combined with classifiers, a supervised learning fault diagnosis method for multilevel converters is constructed. The proposed AIRMAE is able to investigate low-dimensional representations from high-dimensional data manifolds while preserving sufficient valid information. Based on a five-level nested neutral-point piloted converter, the effectiveness of the proposed method is verified by experiments. The performance is superior to that of existing state-of-the-art fault diagnosis methods, with high detection accuracy, good stability, and strong waveform reconstruction ability.

Sparse feature selection via fast embedding spectral analysis

Feature selection has been a research hotspot in many fields. Models based on graph learning are currently the most popular approaches. However, the sparsity of most models is not strong, and graph learning for pair-sample evaluation takes a lot of time. ℓ2,1-norm regularization is the sparsity strategy adopted in most sparse models at present since the convex function is easy to solve. Nevertheless, the sparsity of ℓ2,1-norm is insufficient, and there exist parameter adjustment problems. ℓ2,0-norm is a better choice, which can strengthen the sparse constraints of the subspace. In this paper, the Sparse feature selection via Fast Embedding Spectral Analysis (SFESA) is proposed.Firstly, an adaptive anchor nearest neighbor graph is constructed to avoid the high time cost of learning pairwise nearest neighbor graphs to a certain extent. The low-dimensional embedding of data manifold structure is maintained by performing spectral analysis for the constructed graph. Secondly, the projected data is approximated to the low-dimensional embedding structure via a regularization term. Finally, ℓ2,0-norm is employed to constrain the projection matrix to enhance the subspace sparsity. Furthermore, a fast iterative algorithm is presented to solve this non-convex optimization problem. Extensive experiments on multiple public datasets show that SFESA can obtain excellent performance in less time.

Side effects of learning from low-dimensional data embedded in a Euclidean space

The low-dimensional manifold hypothesis posits that the data found in many applications, such as those involving natural images, lie (approximately) on low-dimensional manifolds embedded in a high-dimensional Euclidean space. In this setting, a typical neural network defines a function that takes a finite number of vectors in the embedding space as input. However, one often needs to consider evaluating the optimized network at points outside the training distribution. This paper considers the case in which the training data are distributed in a linear subspace of $${\mathbb {R}}^d$$ . We derive estimates on the variation of the learning function, defined by a neural network, in the direction transversal to the subspace. We study the potential regularization effects associated with the network’s depth and noise in the codimension of the data manifold. We also present additional side effects in training due to the presence of noise.

Point pattern analysis and classification on compact two-point homogeneous spaces evolving time.

This paper introduces a new modeling framework for the statistical analysis of point patterns on a manifold defined by a connected and compact two-point homogeneous space, including the special case of the sphere. The presented approach is based on temporal Cox processes driven by a -valued log-intensity. Different aggregation schemes on the manifold of the spatiotemporal point-referenced data are implemented in terms of the time-varying discrete Jacobi polynomial transform of the log-risk process. The n-dimensional microscale point pattern evolution in time at different manifold spatial scales is then characterized from such a transform. The simulation study undertaken illustrates the construction of spherical point process models displaying aggregation at low Legendre polynomial transform frequencies (large scale), while regularity is observed at high frequencies (small scale). K-function analysis supports these results under temporal short, intermediate and long range dependence of the log-risk process.

Curvature-Adaptive Meta-Learning for Fast Adaptation to Manifold Data.

Meta-learning methods are shown to be effective in quickly adapting a model to novel tasks. Most existing meta-learning methods represent data and carry out fast adaptation in euclidean space. In fact, data of real-world applications usually resides in complex and various Riemannian manifolds. In this paper, we propose a curvature-adaptive meta-learning method that achieves fast adaptation to manifold data by producing suitable curvature. Specifically, we represent data in the product manifold of multiple constant curvature spaces and build a product manifold neural network as the base-learner. In this way, our method is capable of encoding complex manifold data into discriminative and generic representations. Then, we introduce curvature generation and curvature updating schemes, through which suitable product manifolds for various forms of data manifolds are constructed via few optimization steps. The curvature generation scheme identifies task-specific curvature initialization, leading to a shorter optimization trajectory. The curvature updating scheme automatically produces appropriate learning rate and search direction for curvature, making a faster and more adaptive optimization paradigm compared to hand-designed optimization schemes. We evaluate our method on a broad set of problems including few-shot classification, few-shot regression, and reinforcement learning tasks. Experimental results show that our method achieves substantial improvements as compared to meta-learning methods ignoring the geometry of the underlying space.

SpectralMAP: Approximating Data Manifold With Spectral Decomposition

Dimensionality reduction is widely used to visualize complex high-dimensional data. This study presents a novel method for effective data visualization. Previous methods depend on local distance measurements for data manifold approximation. This leads to unreliable results when a data manifold locally oscillates because of some undesirable effects, such as noise effects. In this study, we overcome this limitation by introducing a dual approximation of a data manifold. We roughly approximate a data manifold with a neighborhood graph and prune it with a global filter. This dual scheme results in local oscillation robustness and yields effective visualization with explicit global preservation. We consider a global filter based on principal component analysis frameworks and derive it with the spectral information of the original high-dimensional data. Finally, we experiment with multiple datasets to verify our method, compare its performance to that of state-of-the-art methods, and confirm the effectiveness of our novelty and results.

A Local Iterative Approach for the Extraction of 2D Manifolds from Strongly Curved and Folded Thin-Layer Structures.

Ridge surfaces represent important features for the analysis of 3-dimensional (3D) datasets in diverse applications and are often derived from varying underlying data including flow fields, geological fault data, and point data, but they can also be present in the original scalar images acquired using a plethora of imaging techniques. Our work is motivated by the analysis of image data acquired using micro-computed tomography ([Formula: see text]) of ancient, rolled and folded thin-layer structures such as papyrus, parchment, and paper as well as silver and lead sheets. From these documents we know that they are 2-dimensional (2D) in nature. Hence, we are particularly interested in reconstructing 2D manifolds that approximate the document's structure. The image data from which we want to reconstruct the 2D manifolds are often very noisy and represent folded, densely-layered structures with many artifacts, such as ruptures or layer splitting and merging. Previous ridge-surface extraction methods fail to extract the desired 2D manifold for such challenging data. We have therefore developed a novel method to extract 2D manifolds. The proposed method uses a local fast marching scheme in combination with a separation of the region covered by fast marching into two sub-regions. The 2D manifold of interest is then extracted as the surface separating the two sub-regions. The local scheme can be applied for both automatic propagation as well as interactive analysis. We demonstrate the applicability and robustness of our method on both artificial data as well as real-world data including folded silver and papyrus sheets.

Wavelet Transform-Assisted Adaptive Generative Modeling for Colorization

Unsupervised deep learning has recently demonstrated the promise to produce high-quality samples. While it has tremendous potential to promote the image colorization task, the performance is limited owing to the high-dimension of data manifold and model capability. This study presents a novel scheme that exploiting the score-based generative model in wavelet domain to address the issue. By taking advantage of the multi-scale and multi-channel representation via wavelet transform, the proposed model learns the priors from stacked wavelet coefficient components, thus learns the image characteristics under coarse and detail frequency spectrums jointly and effectively. Moreover, such a highly flexible generative model without adversarial optimization can execute colorization tasks better under dual consistency terms in wavelet domain, namely data-consistency and structure-consistency. Specifically, in the training phase, a set of multi-channel tensors consisting of wavelet coefficients are used as the input to train the network by denoising score matching. In the test phase, samples are iteratively generated via annealed Langevin dynamics with data and structure consistencies. Experiments demonstrated remarkable improvements of the proposed model on colorization quality, particularly on colorization robustness and diversity.

Multi-modal feature selection with self-expression topological manifold for end-stage renal disease associated with mild cognitive impairment.

Effectively selecting discriminative brain regions in multi-modal neuroimages is one of the effective means to reveal the neuropathological mechanism of end-stage renal disease associated with mild cognitive impairment (ESRDaMCI). Existing multi-modal feature selection methods usually depend on the Euclidean distance to measure the similarity between data, which tends to ignore the implied data manifold. A self-expression topological manifold based multi-modal feature selection method (SETMFS) is proposed to address this issue employing self-expression topological manifold. First, a dynamic brain functional network is established using functional magnetic resonance imaging (fMRI), after which the betweenness centrality is extracted. The feature matrix of fMRI is constructed based on this centrality measure. Second, the feature matrix of arterial spin labeling (ASL) is constructed by extracting the cerebral blood flow (CBF). Then, the topological relationship matrices are constructed by calculating the topological relationship between each data point in the two feature matrices to measure the intrinsic similarity between the features, respectively. Subsequently, the graph regularization is utilized to embed the self-expression model into topological manifold learning to identify the linear self-expression of the features. Finally, the selected well-represented feature vectors are fed into a multicore support vector machine (MKSVM) for classification. The experimental results show that the classification performance of SETMFS is significantly superior to several state-of-the-art feature selection methods, especially its classification accuracy reaches 86.10%, which is at least 4.34% higher than other comparable methods. This method fully considers the topological correlation between the multi-modal features and provides a reference for ESRDaMCI auxiliary diagnosis.

Kernelized transformed subspace clustering with geometric weights for non-linear manifolds

The naive assumption of subspace clustering is that the data should be separable into separate subspaces. Another consideration of the conventional subspace clustering methods is the linear manifolds. What if, the data doesn’t hold this assumption? We propose a novel subspace clustering framework that works even if the raw data is not separable into separate subspaces. It also generalizes it for non-linear manifolds. To achieve the intended goal, we embed subspace clustering techniques into kernelized transform learning with weighting matrix regularization which accounts for nonlinearity. For the weighting matrix, we use similarity between tangent spaces on data manifolds for local structure and euclidean distances for capturing global geometric structure. The complete optimization problem is solved using alternate minimization. The weighted norm regularization is solved by designing a fixed-point continuation algorithm to obtain an approximate closed solution. To test the performance of the proposed framework, we provide the experimental results on handwritten digits clustering, face image clustering, and motion segmentation. The superiority of the results proves the effectiveness of the weighting matrix in kernelized transformed subspace clustering formulation.

Analysis of UMAP, the method for reducing the dimensionality of initial data in machine learning for the purpose of failure prediction in a motive power service

Aim.Feature transformation is one of the stages of machine learning applicationthat has a significant effect on the quality of regression models. The paper aims to developcriteria for evaluating the quality of data dimensionality reduction at the stage of featuretransformation and adaptation of the UMAP method to the problem of prediction of the numberof days to failure in the locomotives of JSC RZD.Methods.The data transformation methodsare divided into two groups, those that attempt to preserve the global data structure, andthose that attempt to preserve the distances between points. The paper examines in detail theUMAP no-linear method of dimensionality reduction, whose low-dimensional data presentationis based on a transformation of a nearest neighbour graph retaining the data structure. Thestructure of the initial data manifold is examined using topological data analysis and simplifiedfuzzy set construction methods.Results.The analysis of UMAP theory conducted in theRussian language for the first time enabled a substantiated identification of the three primaryparameters of the method, whose variation significantly affects the type of data obtained asthe result of a transformation. In particular, that pertains to the quality of class separationover a two-dimensional space. Additionally, the characteristics of the input set of parameterswere identified that affect the UMAP results. Practical results of UMAP application weredemonstrated. Intermediate results included a list of nearest neighbours, a weighted graph of nearest neighbours. The fundamental result is a low-dimensional data representation (outof 44 initial measurements) over a two-dimensional space with class separation, which isconfirmed both by calculations, and visually.Conclusions.It was identified that UMAP is anefficient and substantiated method of dimensionality reduction that allows – through parametervariation – transforming data in such a way as to improve the quality of data submitted tomachine learning models by the criterion of “evident class separation”. The transformation is anintermediate stage of data preparation for regression model application, and class separationwas performed for the purpose of eliminating the probability of gross regression errors.

Locally alignment based manifold learning for simultaneous feature selection and extraction in classification problems

Dimensionality reduction is an important step in increasing the performance of machine learning algorithms while decreasing the processing time. From feature reduction approaches, feature extraction is aimed to achieve better data representation while feature selection is meant to discard redundant features. The main contribution of this manuscript is to propose a hybrid feature selection and extraction approach which performs these tasks in a unified framework, simultaneously. Also, the proposed approach aims to maintain the manifold of data using locally alignment constrains. The main goal of these suggestions is to discriminate different classes while reducing the redundancy of the samples specially for the imbalanced data classification problems. The proposed approach optimizes a hybrid objective function that tries to both increase the between class discrimination and decrease the within class distribution while minimizing the information loss. Also, we have embedded a weighting factor into the objective function to achieve a basis for feature importance measurement and feature selection. To evaluate the proposed approach, a multiclass linear SVM classifier is applied on the reduced data in a k-fold cross-validation scheme, and the accuracy, as well as F-score, is used as the performance measure. Comparisons of the proposed method with some recent approaches on 12 datasets of UCI, show the superiority of the proposed method over previously proposed approaches. Also, Friedman and Nemenyi tests are applied which show significant improvement in the proposed approach.

Joint Sparse Locality-Aware Regression for Robust Discriminative Learning.

With the dramatic increase of dimensions in the data representation, extracting latent low-dimensional features becomes of the utmost importance for efficient classification. Aiming at the problems of weakly discriminating marginal representation and difficulty in revealing the data manifold structure in most of the existing linear discriminant methods, we propose a more powerful discriminant feature extraction framework, namely, joint sparse locality-aware regression (JSLAR). In our model, we formulate a new strategy induced by the nonsquared L2 norm for enhancing the local intraclass compactness of the data manifold, which can achieve the joint learning of the locality-aware graph structure and the desirable projection matrix. Besides, we formulate a weighted retargeted regression to perform the marginal representation learning adaptively instead of using the general average interclass margin. To alleviate the disturbance of outliers and prevent overfitting, we measure the regression term and locality-aware term together with the regularization term by forcing the row sparsity with the joint L2,1 norms. Then, we derive an effective iterative algorithm for solving the proposed model. The experimental results over a range of benchmark databases demonstrate that the proposed JSLAR outperforms some state-of-the-art approaches.

Multiview Graph Restricted Boltzmann Machines.

Recently, the restricted Boltzmann machine (RBM) has aroused considerable interest in the multiview learning field. Although effectiveness is observed, like many existing multiview learning models, multiview RBM ignores the local manifold structure of multiview data. In this article, we first propose a novel graph RBM model, which preserves the data manifold structure and is amenable to Gibbs sampling. Then, we develop a multiview graph RBM model on the basis of the graph RBM, which performs local structural learning and multiview representation learning simultaneously. The proposed multiview model has the following merits: 1) it preserves the data manifold structure for multiview classification and 2) it performs view-consistent representation learning and view-specific representation learning simultaneously. The experimental results show that the proposed multiview model outperforms other state-of-the-art multiview classification algorithms.

Subdomain Adaptation With Manifolds Discrepancy Alignment.

Reducing domain divergence is a key step in transfer learning. Existing works focus on the minimization of global domain divergence. However, two domains may consist of several shared subdomains, and differ from each other in each subdomain. In this article, we take the local divergence of subdomains into account in transfer. Specifically, we propose to use the low-dimensional manifold to represent the subdomain, and align the local data distribution discrepancy in each manifold across domains. A manifold maximum mean discrepancy (M3D) is developed to measure the local distribution discrepancy in each manifold. We then propose a general framework, called transfer with manifolds discrepancy alignment (TMDA), to couple the discovery of data manifolds with the minimization of M3D. We instantiate TMDA in the subspace learning case considering both the linear and nonlinear mappings. We also instantiate TMDA in the deep learning framework. Experimental studies show that TMDA is a promising method for various transfer learning tasks.

A semi-supervised label distribution learning model with label correlations and data manifold exploration

Label distribution learning is a novel machine learning paradigm to deal with label ambiguity, which is the generalization of the traditional single-label learning and multi-label learning paradigms. Though label distribution learning has attracted a lot of attentions recently, data sets with label distributions rather than logical labels have always been scarce and most of the existing models put emphasis mainly on the supervised learning, neglecting the utilization of unlabeled samples. In this paper, we propose a semi-supervised Label Distribution Learning model with label Correlations and data Manifold exploration (sLDLCM). On one hand, sLDLCM is a semi-supervised extension of existing label distribution learning models, which can effectively make use of both labeled and unlabeled data for better capturing the underlying data properties; on the other hand, sLDLCM jointly estimates the label distributions of unlabeled samples and the other model variables. Besides, both label correlations and local data manifold are explored in sLDLCM. Extensive experiments are conducted on six real-world data sets including two facial expression, one movie rating, two bioinformatics and one visual sentiment analysis data sets. Comparative studies demonstrate that the proposed sLDLCM model achieves better performance the state-of-the-arts in terms of five evaluation metrics.

Comparing the latent space of generative models

Different encodings of datapoints in the latent space of latent-vector generative models may result in more or less effective and disentangled characterizations of the different explanatory factors of variation behind the data. Many works have been recently devoted to the exploration of the latent space of specific models, mostly focused on the study of how features are disentangled and of how trajectories producing desired alterations of data in the visible space can be found. In this work we address the more general problem of comparing the latent spaces of different models, looking for transformations between them. We confined the investigation to the familiar and largely investigated case of generative models for the data manifold of human faces. The surprising, preliminary result reported in this article is that (provided models have not been taught or explicitly conceived to act differently) a simple linear mapping is enough to pass from a latent space to another while preserving most of the information. This is full of consequences for representation learning, potentially paving the way to the transformation of editing trajectories from one space to another, or the adaptation of disentanglement techniques between different generative domains.