Data Manifold Research Articles

Wasserstein-Based Projections with Applications to Inverse Problems

Inverse problems consist of recovering a signal from a collection of noisy measurements. These are typically cast as optimization problems, with classic approaches using a data fidelity term and an analytic regularizer that stabilizes recovery. Recent Plug-and-Play (PnP) works propose replacing the operator for analytic regularization in optimization methods by a data-driven denoiser. These schemes obtain state of the art results, but at the cost of limited theoretical guarantees. To bridge this gap, we present a new algorithm that takes samples from the manifold of true data as input and outputs an approximation of the projection operator onto this manifold. Under standard assumptions, we prove this algorithm generates a learned operator, called Wasserstein-based projection (WP), that approximates the true projection with high probability. Thus, WPs can be inserted into optimization methods in the same manner as PnP, but now with theoretical guarantees. Provided numerical examples show WPs obtain state of the art results for unsupervised PnP signal recovery.

Generative Adversarial learning with Negative Data Augmentation for Semi-supervised Text Classification

In recent years, semi-supervised generative adversarial networks (SS-GANs) models such as GAN-BERT have achieved promising results on the text classification task. One of the techniques used in these models to mitigate the generator from mode collapse is feature matching (FM). Although FM addresses some of the critical issues of SS-GANs, these models still suffer from mode collapse with missing coverage outside the data manifold. Moreover, FM loosely tries to match the distribution between the real data and the fake generated samples. By doing this, the generator can generate fake samples inside high-density regions in the data manifold, where the discriminator learns to misclassify them as out-of-data-manifold regions. In this work, we employ the negative data augmentation (NDA) technique, for the first time in text classification, to alleviate the mentioned problems. NDA is a unique way of producing out-of-distribution fake examples by applying mixup transformation on the fake samples and augmented real data. In our new model (NDA-GAN), we produce NDA samples by combining the generator's output with the contextual representation of the real data. As a result of the mixing, NDA samples are less likely to place in the high-density regions, and due to blending with real data representations, these samples reasonably preserve a close distance to the data manifold. Consequently, the NDA samples increase the discriminator's power to find the optimal decision boundary. Our experimental results demonstrate that the negative augmented samples improve the overall accuracy of our proposed model and make it more confident when detecting out-of-distribution samples.

Informative knowledge distillation for image anomaly segmentation

Unsupervised anomaly segmentation methods based on knowledge distillation have recently been developed and have shown superior segmentation performance. However, little attention has been paid to the overfitting problem caused by the inconsistency between the capacity of a neural network and the amount of knowledge in this scheme. This study proposes a novel method called informative knowledge distillation (IKD) to address the overfitting problem by distilling informative knowledge and offering a strong supervisory signal. Technically, a novel context similarity loss method is proposed to capture context information from normal data manifolds. In addition, a novel adaptive hard sample mining method is proposed to encourage more attention on hard samples with valuable information. With IKD, informative knowledge can be distilled such that the overfitting problem can be effectively mitigated, and the performance can be further increased. The proposed method achieved better results on several categories of the well-known MVTec AD dataset than state-of-the-art methods in terms of AU-ROC, achieving 97.81% overall in 15 categories. Extensive experiments on ablation have also been conducted to demonstrate the effectiveness of IKD in alleviating the overfitting problem.

Atlas generative models and geodesic interpolation

Generative neural networks have a well recognized ability to estimate underlying manifold structure of high dimensional data. However, if a single latent space is used, it is not possible to faithfully represent a manifold with topology different from Euclidean space. In this work we define the general class of Atlas Generative Models (AGMs), models with hybrid discrete-continuous latent space that estimate an atlas on the underlying data manifold together with a partition of unity on the data space. We identify existing examples of models from various popular generative paradigms that fit into this class. Due to the atlas interpretation, ideas from non-linear latent space analysis and statistics, e.g. geodesic interpolation, which has previously only been investigated for models with simply connected latent spaces, may be extended to the entire class of AGMs in a natural way. We exemplify this by generalizing an algorithm for graph based geodesic interpolation to the setting of AGMs, and verify its performance experimentally.

Joint Characterization of Spatiotemporal Data Manifolds

Modeling spatiotemporal data can be a challenge due to the plethora of processes, both independent and interacting, which may or may not contribute to the measurements. Characterization can be considered a complement to modeling by helping guide assumptions about generative processes and their representation in the data. For high-D signals, Dimensionality Reduction (DR) is a frequently implemented type of characterization designed to mitigate the effects of the so-called “curse of dimensionality”. For decades, Principal Component (PC) and Empirical Orthogonal Function (EOF) analysis has been used as a linear, invertible approach to dimensionality reduction and spatiotemporal analysis. Recent years have seen the additional development of a suite of nonlinear DR algorithms, frequently categorized as “manifold learning”. Here, we explore the idea of joint characterization of spatiotemporal data manifolds using the PC/EOF approach alongside two nonlinear DR approaches: Laplacian Eigenmaps (LE) and t-distributed Stochastic Neighbor Embedding (t-SNE). Starting with a synthetic example and progressing to global, regional, and field scale spatiotemporal datasets spanning roughly 5 orders of spatial magnitude and 2 orders of temporal magnitude, we show these three DR approaches can yield complementary information about the topology of spatiotemporal data manifolds. Compared to the PC/EOF projections, the nonlinear DR approaches yield more compact manifolds with decreased ambiguity in temporal endmembers (LE) and/or in spatiotemporal clustering (t-SNE), compared to the relatively diffuse temporal feature space produced by the PC/EOF approach. However, these properties are compensated by the greater interpretability of PCs and EOFs than of the LE or t-SNE dimensions, as well as significantly lower computational demand and diminished sensitivity to spatial aliasing for PCs/EOFs than LE or t-SNE. Taken together, we find the joint characterization using the three complementary DR approaches capable of providing substantially greater insight about the generative processes represented in spatiotemporal datasets than is possible using any single approach alone. This parsimonious, complementary characterization of both local manifold structure and global variance can advance remote sensing time series analysis by providing important context to constrain and guide design of effective spatiotemporal models.

Generative models with kernel distance in data space

Generative models dealing with modeling a joint data distribution are generally autoencoder, or GAN based. Both have pros and cons, generating blurry images or being unstable in training. We propose a new generative model resembling a classical GAN transforming Gaussian noise into data space without adversarial optimization. Training of the proposed model is a two-step procedure. First, we train an autoencoder-based architecture to model a data manifold. Second, we use the Latent Trick to map Gaussian noise into the autoencoder’s latent space. The resulting Latent Cramer-Wold (LCW) generator achieves competitive generative scores. Elimination of adversarial training and replacing the discriminator with kernel methods results in a stable training procedure that is not prone to mode collapse. We also show that the introduced Latent Trick can improve the generative capabilities of other latent-based models. We validate the model on standard benchmarks and compare it to different approaches.

Adversarial Autoencoder Based Feature Learning for Fault Detection in Industrial Processes

Deep learning has recently emerged as a promising method for nonlinear process monitoring. However, ensuring that the features from process variables have representative information of the high-dimensional process data remains a challenge. In this study, we propose an adversarial autoencoder (AAE) based process monitoring system. AAE which combines the advantages of a variational autoencoder and a generative adversarial network enables the generation of features that follow the designed prior distribution. By employing the AAE model, features that have informative manifolds of the original data are obtained. These features are used for constructing and monitoring statistics and improve the stability and reliability of fault detection. Extracted features help calculate the degree of abnormalities in process variables more robustly and indicate the type of fault information they imply. Finally, our proposed method is testified using the Tennessee Eastman benchmark process in terms of fault detection rate, false alarm rate, and fault detection delays.

Data Augmentation for Audio-Visual Emotion Recognition with an Efficient Multimodal Conditional GAN

Audio-visual emotion recognition is the research of identifying human emotional states by combining the audio modality and the visual modality simultaneously, which plays an important role in intelligent human-machine interactions. With the help of deep learning, previous works have made great progress for audio-visual emotion recognition. However, these deep learning methods often require a large amount of data for training. In reality, data acquisition is difficult and expensive, especially for the multimodal data with different modalities. As a result, the training data may be in the low-data regime, which cannot be effectively used for deep learning. In addition, class imbalance may occur in the emotional data, which can further degrade the performance of audio-visual emotion recognition. To address these problems, we propose an efficient data augmentation framework by designing a multimodal conditional generative adversarial network (GAN) for audio-visual emotion recognition. Specifically, we design generators and discriminators for audio and visual modalities. The category information is used as their shared input to make sure our GAN can generate fake data of different categories. In addition, the high dependence between the audio modality and the visual modality in the generated multimodal data is modeled based on Hirschfeld-Gebelein-Rényi (HGR) maximal correlation. In this way, we relate different modalities in the generated data to approximate the real data. Then, the generated data are used to augment our data manifold. We further apply our approach to deal with the problem of class imbalance. To the best of our knowledge, this is the first work to propose a data augmentation strategy with a multimodal conditional GAN for audio-visual emotion recognition. We conduct a series of experiments on three public multimodal datasets, including eNTERFACE’05, RAVDESS, and CMEW. The results indicate that our multimodal conditional GAN has high effectiveness for data augmentation of audio-visual emotion recognition.

Geodesic Learning With Uniform Interpolation on Data Manifold

Recently with the development of deep learning on data representation and generation, how to sampling on a data manifold becomes a crucial problem for research. In this paper, we propose a method to learn a minimizing geodesic within a data manifold. Along the learned geodesic, our method is able to generate high-quality uniform interpolations with the shortest path between two given data samples. Specifically, we use an autoencoder network to map data samples into the latent space and perform interpolation in the latent space via an interpolation network. We add prior geometric information to regularize our autoencoder for a flat latent embedding. The Riemannian metric on the data manifold is induced by the canonical metric in the Euclidean space in which the data manifold is isometrically immersed. Based on this defined Riemannian metric, we introduce a constant-speed loss and a minimizing geodesic loss to regularize the interpolation network to generate uniform interpolations along the learned geodesic on the manifold. We provide a theoretical analysis of our model and use image interpolation as an example to demonstrate the effectiveness of our method.

Ensemble Learning With Manifold-Based Data Splitting for Noisy Label Correction

Label noise in training data can significantly degrade a model’s generalization performance for supervised learning tasks. Here we focus on the problem that noisy labels are primarily caused by mislabeled confusing samples, which tend to be concentrated near decision boundaries rather than uniformly distributed, and whose features should be equivocal. To address the problem, we propose an ensemble learning method to correct noisy labels by exploiting the local structures of feature manifolds. Different from typical ensemble strategies that increase the prediction diversity among sub-models via certain loss terms, our method trains sub-models on disjoint subsets, each being a union of randomly selected seed samples’ nearest-neighbors of the same class on the data manifold. As a result, only a limited number of sub-models will be affected by locally-concentrated noisy labels, and each sub-model can learn a coarse representation of the data manifold along with a corresponding graph. The constructed graphs are used to suggest a set of label correction candidates, and accordingly, our method determines label correction results by majority decisions. Our experiments on real-world noisy label datasets demonstrate the superiority of the proposed method over existing state-of-the-arts.

Convolutional Graph Thermography for Subsurface Defect Detection in Polymer Composites

Infrared thermography for quality assessment of polymer composites has gained increasing attention with the development of various thermographic data analysis methods. However, low-quality thermal images containing noise and inhomogeneous backgrounds restrict the defect detection performance of these methods. In this work, a convolutional graph thermography (CGT) method is proposed for subsurface defect detection in polymer composites. With a developed convolutional autoencoder thermogram enhancer, the noise and inhomogeneous background of the thermal image can be reduced. Subsequently, a graph-based dimensionality reduction method performs feature extraction on the enhanced thermograms. Consequently, the low-dimensional intrinsic manifolds of the high-dimensional thermographic data are characterized to highlight the defects visually. Two case studies on carbon fiber reinforced polymers demonstrated the performance of the proposed method.

A Vertex-Directed Evolutionary Algorithm for Multiobjective Endmember Estimation

Hyperspectral unmixing including endmember extration and abundance estimation has been investigated successively in recent years due to increasingly hyperspectral processing requirements. As one type of decision after solution paradigm, multiobjective endmember estimation method is able to obtain a set of Pareto optimal solutions, thus providing a wealth of information to determine the most representative endmembers. In addition, multiobjective optimization methods also have the characteristics of flexible modeling, excellent global convergence and commendable adaptability, etc. However, the evolutionary algorithms designed for this kind of method generally use little spatial-spectral information of the hyperspectral image. In this paper, we delve into the memetic strategy by exploiting the topological structure of hyperspectral data in the high-dimensional space to establish a vertex-directed multiobjective endmember estimation method, termed VD-MoEE. According to the frequently used linear mixture model, endmembers of hyperspectral images are generally distributed at the vertices of hyperspectral data manifold. Therefore, we design a vertex-directed local search operator to guide the search direction of the individuals in evolutionary algorithms. Experimental results on synthetic as well as real data sets demonstrated that the proposed VD-MOEE is able to achieve appealing performance in terms of the solution selecting and accuracy in comparison with several classic and state-of-the-art endmember estimation methods.

Attention Regularized Laplace Graph for Domain Adaptation.

In leveraging manifold learning in domain adaptation (DA), graph embedding-based DA methods have shown their effectiveness in preserving data manifold through the Laplace graph. However, current graph embedding DA methods suffer from two issues: 1). they are only concerned with preservation of the underlying data structures in the embedding and ignore sub-domain adaptation, which requires taking into account intra-class similarity and inter-class dissimilarity, thereby leading to negative transfer; 2). manifold learning is proposed across different feature/label spaces separately, thereby hindering unified comprehensive manifold learning. In this paper, starting from our previous DGA-DA, we propose a novel DA method, namely A ttention R egularized Laplace G raph-based D omain A daptation (ARG-DA), to remedy the aforementioned issues. Specifically, by weighting the importance across different sub-domain adaptation tasks, we propose the A ttention R egularized Laplace Graph for class aware DA, thereby generating the attention regularized DA. Furthermore, using a specifically designed FEEL strategy, our approach dynamically unifies alignment of the manifold structures across different feature/label spaces, thus leading to comprehensive manifold learning. Comprehensive experiments are carried out to verify the effectiveness of the proposed DA method, which consistently outperforms the state of the art DA methods on 7 standard DA benchmarks, i.e., 37 cross-domain image classification tasks including object, face, and digit images. An in-depth analysis of the proposed DA method is also discussed, including sensitivity, convergence, and robustness.

ScoreMix: A Scalable Augmentation Strategy for Training GANs With Limited Data.

Generative Adversarial Networks (GANs) typically suffer from overfitting when limited training data is available. To facilitate GAN training, current methods propose to use data-specific augmentation techniques. Despite the effectiveness, it is difficult for these methods to scale to practical applications. In this article, we present ScoreMix, a novel and scalable data augmentation approach for various image synthesis tasks. We first produce augmented samples using the convex combinations of the real samples. Then, we optimize the augmented samples by minimizing the norms of the data scores, i.e., the gradients of the log-density functions. This procedure enforces the augmented samples close to the data manifold. To estimate the scores, we train a deep estimation network with multi-scale score matching. For different image synthesis tasks, we train the score estimation network using different data. We do not require the tuning of the hyperparameters or modifications to the network architecture. The ScoreMix method effectively increases the diversity of data and reduces the overfitting problem. Moreover, it can be easily incorporated into existing GAN models with minor modifications. Experimental results on numerous tasks demonstrate that GAN models equipped with the ScoreMix method achieve significant improvements.

Iterative Spatial-Spectral Training Sample Augmentation for Effective Hyperspectral Image Classification

Factors such as insufficient training samples, high-dimensional data features, and unbalanced data classes can degrade the accuracy of hyperspectral classification. To this end, this paper proposes an iterative spatial-spectral training sample augmentation (ITSA) algorithm and a new classification model incorporating maximum marginal projection (ITSA-MMP). First, ITSA iteratively augments samples by a similar region clustering strategy (SRCS) integrating spatial-spectral metric. Then, box-plot for representative sample selection (BPRSS) is adopted to screen optimal samples for the final augmented sample set (ASS). Next, based on the ASS, MMP projects the hyperspectral image into a low-dimensional subspace to explore the local structure of the data manifold and improve the inter-class separability of the data. Finally, the MMP-reduced data is classified by support vector machine. Experiments on two real hyperspectral datasets validate that ITSA-MMP can effectively increase the training sample set especially for small initial sample set and unbalanced dataset, and obtain a higher classification accuracy.

LAMBO: Landmarks Augmentation With Manifold-Barycentric Oversampling

The training of Generative Adversarial Networks (GANs) requires a large amount of data, stimulating the development of new augmentation methods to alleviate the challenge. Oftentimes, these methods either fail to produce enough new data or expand the dataset beyond the original manifold. In this paper, we propose a new augmentation method that guarantees to keep the new data within the original data manifold thanks to the optimal transport theory. The proposed algorithm finds cliques in the nearest-neighbors graph and, at each sampling iteration, randomly draws one clique to compute the Wasserstein barycenter with random uniform weights. These barycenters then become the new natural-looking elements that one could add to the dataset. We apply this approach to the problem of landmarks detection and augment the available annotation in both unpaired and in semi-supervised scenarios. Additionally, the idea is validated on cardiac data for the task of medical segmentation. Our approach reduces the overfitting and improves the quality metrics beyond the original data outcome and beyond the result obtained with popular modern augmentation methods.

Topology Structure Analysis of High Dimensional Dataset by Flattening Deformation of Data Manifold

A new analysis method for high dimensional sets is proposed by autonomous deforming of data manifolds. The deformation is guided by two kinds of virtual interactions between data points. The flattening of data manifold is achieved under the elastic and repelling interactions, meanwhile the topological structure of the manifold is preserved. The proposed method provides a novel geometric viewpoint on high-dimensional data analysis. Experimental results prove the effectiveness of the proposed method in dataset structure analysis.

Subspace clustering via adaptive least square regression with smooth affinities

It has been proved that subspace clustering algorithms with weighted norm regularizers usually outperform their special cases with non-weighted regularizers. However, it is difficult to design suitable weighted regularizers for different types of subspace clustering algorithms. In this paper, we firstly provide an adaptive weighted norm regularizer construction strategy by using the recently proposed adaptive graph construction method. Secondly, we give a proposition that the affinities obtained by the adaptive graph construction method should be changed smoothly on the original data manifold. Hence, we devise a graph-constraint for the obtained affinity matrix. Thirdly, by integrating the above two techniques and least square regression (LSR) algorithm together, we design a new subspace clustering algorithm, called adaptive least square regression with smooth affinities (ALSR). For solving ALSR problem, we present an optimization algorithm whose computation burden and convergence are analyzed. Finally, plentiful experiments conducted on different types databases show the superiorities of ALSR.

Anomaly Detection for Medical Images Using Self-Supervised and Translation-Consistent Features.

As the labeled anomalous medical images are usually difficult to acquire, especially for rare diseases, the deep learning based methods, which heavily rely on the large amount of labeled data, cannot yield a satisfactory performance. Compared to the anomalous data, the normal images without the need of lesion annotation are much easier to collect. In this paper, we propose an anomaly detection framework, namely [Formula: see text], extracting [Formula: see text]elf-supervised and tr [Formula: see text]ns [Formula: see text]ation-consistent features for [Formula: see text]nomaly [Formula: see text]etection. The proposed SALAD is a reconstruction-based method, which learns the manifold of normal data through an encode-and-reconstruct translation between image and latent spaces. In particular, two constraints (i.e., structure similarity loss and center constraint loss) are proposed to regulate the cross-space (i.e., image and feature) translation, which enforce the model to learn translation-consistent and representative features from the normal data. Furthermore, a self-supervised learning module is engaged into our framework to further boost the anomaly detection accuracy by deeply exploiting useful information from the raw normal data. An anomaly score, as a measure to separate the anomalous data from the healthy ones, is constructed based on the learned self-supervised-and-translation-consistent features. Extensive experiments are conducted on optical coherence tomography (OCT) and chest X-ray datasets. The experimental results demonstrate the effectiveness of our approach.

Distance Preserving Model Order Reduction of Graph-Laplacians and Cluster Analysis

Graph-Laplacians and their spectral embeddings play an important role in multiple areas of machine learning. This paper is focused on graph-Laplacian dimension reduction for the spectral clustering of data as a primary application, however, it can also be applied in data mining, data manifold learning, etc. Spectral embedding provides a low-dimensional parametrization of the data manifold which makes the subsequent task (e.g., clustering with k-means or any of its approximations) much easier. However, despite reducing the dimensionality of data, the overall computational cost may still be prohibitive for large data sets due to two factors. First, computing the partial eigendecomposition of the graph-Laplacian typically requires a large Krylov subspace. Second, after the spectral embedding is complete, one still has to operate with the same number of data points, which may ruin the efficiency of the approach. For example, clustering of the embedded data is typically performed with various relaxations of k-means which computational cost scales poorly with respect to the size of data set. Also, they become prone to getting stuck in local minima, so their robustness depends on the choice of initial guess. In this work, we switch the focus from the entire data set to a subset of graph vertices (target subset). We develop two novel algorithms for such low-dimensional representation of the original graph that preserves important global distances between the nodes of the target subset. In particular, it allows to ensure that target subset clustering is consistent with the spectral clustering of the full data set if one would perform such. That is achieved by a properly parametrized reduced-order model (ROM) of the graph-Laplacian that approximates accurately the diffusion transfer function of the original graph for inputs and outputs restricted to the target subset. Working with a small target subset reduces greatly the required dimension of Krylov subspace and allows to exploit the conventional algorithms (like approximations of k-means) in the regimes when they are most robust and efficient. This was verified in the numerical clustering experiments with both synthetic and real data. We also note that our ROM approach can be applied in a purely transfer-function-data-driven way, so it becomes the only feasible option for extremely large graphs that are not directly accessible. There are several uses for our algorithms. First, they can be employed on their own for representative subset clustering in cases when handling the full graph is either infeasible or simply not required. Second, they may be used for quality control. Third, as they drastically reduce the problem size, they enable the application of more sophisticated algorithms for the task under consideration (like more powerful approximations of k-means based on semi-definite programming (SDP) instead of the conventional Lloyd’s algorithm). Finally, they can be used as building blocks of a multi-level divide-and-conquer type algorithm to handle the full graph. The latter will be reported in a separate article.