Linear Autoencoder Research Articles

KGCF: Social relationship-aware graph collaborative filtering for recommendation

With the increasing popularity of recommendation techniques and social networks, social network recommendation has become a significant research field, i.e., predicting a user's preferences based on her or his historical interaction data, because the social relationships of users can not only enrich their interaction information, but also imply the accurate characterization of their preferences. Although there are some existing studies that predict users' preferences by using the characteristics of social relationships, their time complexity and hardware resource consumption are often very expensive, which limits the feasibility in large-scale real-world scenarios. To address these issues, in this paper, we propose a simple and effective K-core Graph Collaborative Filtering (KGCF) model to incorporate the user's social features into a recommendation framework. The user's interaction information is auto-encoded linearly by incorporating the social relationship graph into the user's interaction information and constructing a multi-layer graph filter. The linear autoencoder graph filter alleviates the highly sparse data problem and dramatically reduces the training time and hardware resource consumption. Experimental results on several real-world datasets show the superior effectiveness and spatiotemporal efficiency compared with the existing baselines, especially the training speed is significantly accelerated for large-scale datasets.

LTScoder: Long-Term Time Series Forecasting Based on a Linear Autoencoder Architecture

LTScoder: Long-Term Time Series Forecasting Based on a Linear Autoencoder Architecture

Locality Preserving Projections with Autoencoder

Locality Preserving Projections (LPP) is a popular dimensionality reduction method in the manifold learning field. However, LPP and all its variants only consider the one-way mapping from the high-dimensional space to the low-dimensional space and have no reverse verification, resulting in inaccurate low-dimensional embeddings. In this paper, we propose a new LPP method, called LPPAE (Locality Preserving Projections with Autoencoder), based on the linear Autoencoder. It constructs a two-way mapping: at the encoding stage, the conventional projection of LPP is viewed as a mapping from the high-dimensional space to the low-dimensional space. At the decoding stage, the low-dimensional embeddings are mapped back to the original high-dimensional space. The main contributions of the new method are: (1) This design not only preserves the neighborhood relationship of the data but more importantly, ensures that the low-dimensional embeddings can more accurately ”represent” the original data, thus significantly improving the performance of LPP. Experimental results on Handwritten Alphadigits, COIL-20, Yale, AR datasets show that the recognition rates of LPPAE are 26.06, 10.09, 5.40, and 8.86% higher than those of the original LPP respectively. On the MNIST dataset, compared to some of the latest improvements of LPP, including LPPMDC, LAPP, LPP+TR, and DNLPP, the recognition rate of LPPAE has been improved by 12.50, 38.10, 9.10, and 2.61%, respectively. (2) LPPAE regards the conventional LPP as an encoder, which is a new perspective. The idea of LPPAE can be used as a general framework and then extended to other manifold learning methods, and then a series of new methods can be developed.

Neighborhood preserving embedding with autoencoder

Neighborhood preserving embedding (NPE) is a classical method for dimensionality reduction (DR), and it is a linear version of the locally linear embedding method. However, NPE and all its variants only consider the one-way mapping from high-dimensional space to low-dimensional space. The projected low-dimensional data may not accurately and effectively “represent” the original samples. To address this problem, we improve NPE based on linear autoencoder. The conventional projection of NPE is considered as the encoding stage, and the decoder stage is a reconstruction from the low-dimensional space to the original high-dimensional space, which is the key to maintaining more significant information. Based on this, we propose a new NPE method called NPEAE (neighborhood preserving embedding with autoencoder) in this paper. NPEAE performs excellently in face recognition, handwritten character categorization, object classification, etc. The experiments on MNIST, COIL-20, the Extended Yale B, Olivetti Research Laboratory (ORL), and FERET show that NPEAE has a higher recognition accuracy than other comparative methods.

The dynamics of representation learning in shallow, non-linear autoencoders * * This article is an updated version of: Refinetti M and Goldt S 2022 The dynamics of representation learning in shallow, non-linear autoencoders Proc. 39th Int. Conf. on Machine Learning (Proc. Machine Learning Research vol 162) ed K Chaudhuri, S Jegelka, L Song, C Szepesvari, G Niu and S Sabato

Autoencoders are the simplest neural network for unsupervised learning, and thus an ideal framework for studying feature learning. While a detailed understanding of the dynamics of linear autoencoders has recently been obtained, the study of non-linear autoencoders has been hindered by the technical difficulty of handling training data with non-trivial correlations—a fundamental prerequisite for feature extraction. Here, we study the dynamics of feature learning in non-linear, shallow autoencoders. We derive a set of asymptotically exact equations that describe the generalisation dynamics of autoencoders trained with stochastic gradient descent (SGD) in the limit of high-dimensional inputs. These equations reveal that autoencoders learn the leading principal components of their inputs sequentially. An analysis of the long-time dynamics explains the failure of sigmoidal autoencoders to learn with tied weights, and highlights the importance of training the bias in ReLU autoencoders. Building on previous results for linear networks, we analyse a modification of the vanilla SGD algorithm, which allows learning of the exact principal components. Finally, we show that our equations accurately describe the generalisation dynamics of non-linear autoencoders trained on realistic datasets such as CIFAR10, thus establishing shallow autoencoders as an instance of the recently observed Gaussian universality.

Extraction and analysis of flow features in planar synthetic jets using different machine learning techniques

Synthetic jets are useful fluid devices with several industrial applications. In this study, we use the flow fields generated by two synchronously operating synthetic jets and simulated using direct numerical simulations. These flow fields are characterized by a jet Reynolds number, Re=100, 150, and 200, and a Strouhal number, St=0.03. We benchmark four different dimensionality reduction techniques: (1) higher-order dynamic mode decomposition (HODMD), (2) proper orthogonal decomposition, (3) vector quantization via principal component analysis (VQPCA), and (4) linear autoencoders. These techniques are often used in generating reduced-order models (ROMs). The performances of these techniques are compared (i) in terms of their ability to accurately reconstruct the high-dimensional flow fields from their low-dimensional manifolds and (ii) in terms of their ability to extract meaningful low-dimensional patterns/features/structures that best describe the main dynamics of the synthetic jets. The similarity between the extracted features is also quantitatively assessed with the help of Procrustes analysis, showing how manifolds from different techniques become more similar when a larger number of modes are retained. Accurate reconstruction and model complexity (or interpretability) are often two counter-balancing objectives. In this comparative study, we found that among the four techniques, VQPCA has clear advantages for developing accurate ROMs, while HODMD is useful for understanding the dynamics of synthetic jets, providing additional information that is not readily available with other methods.

Linear local tangent space alignment with autoencoder

Linear local tangent space alignment (LLTSA) is a classical dimensionality reduction method based on manifold. However, LLTSA and all its variants only consider the one-way mapping from high-dimensional space to low-dimensional space. The projected low-dimensional data may not accurately and effectively “represent” the original samples. This paper proposes a novel LLTSA method based on the linear autoencoder called LLTSA-AE (LLTSA with Autoencoder). The proposed LLTSA-AE is divided into two stages. The conventional process of LLTSA is viewed as the encoding stage, and the additional and important decoding stage is used to reconstruct the original data. Thus, LLTSA-AE makes the low-dimensional embedding data “represent” the original data more accurately and effectively. LLTSA-AE gets the recognition rates of 85.10, 67.45, 75.40 and 86.67% on handwritten Alphadigits, FERET, Georgia Tech. and Yale datasets, which are 9.4, 14.03, 7.35 and 12.39% higher than that of the original LLTSA respectively. Compared with some improved methods of LLTSA, it also obtains better performance. For example, on Handwritten Alphadigits dataset, compared with ALLTSA, OLLTSA, PLLTSA and WLLTSA, the recognition rates of LLTSA-AE are improved by 4.77, 3.96, 7.8 and 8.6% respectively. It shows that LLTSA-AE is an effective dimensionality reduction method.

Self-Supervised Learning for 3-D Point Clouds Based on a Masked Linear Autoencoder

Self-Supervised Learning for 3-D Point Clouds Based on a Masked Linear Autoencoder

Matrix Factorization-Based Dimensionality Reduction Algorithms─A Comparative Study on Spectroscopic Profiling Data.

Spectroscopic profiling data used in analytical chemistry can be very high-dimensional. Dimensionality reduction (DR) is an effective way to handle the potential "curse of dimensionality" problem. Among the existing DR algorithms, many can be categorized as a matrix factorization (MF) problem, which decomposes the original data matrix X into the product of a low-dimensional matrix W and a dictionary matrix H. First, this paper provides a theoretical reformulation of relevant DR algorithms under a unified MF perspective, including PCA (principal component analysis), NMF (non-negative matrix factorization), LAE (linear autoencoder), RP (random projection), SRP (sparse random projection), VQ (vector quantization), AA (archetypical analysis), and ICA (independent component analysis). From this perspective, an open-sourced toolkit has been developed to integrate all of the above algorithms with a unified API. Second, we made a comparative study on MF-based DR algorithms. In a case study of TOF (time-of-flight) mass spectra, the eight algorithms extracted three components from the original 27,619 features. The results are compared by a set of DR quality metrics, e.g., reconstruction error, pairwise distance/ranking property, computational cost, local and global structure preservations, etc. Finally, based on the case study result, we summarized guidelines for DR algorithm selection. (1) For reconstruction quality, choose ICA. In the case study, ICA, PCA, and NMF have high reconstruction qualities (reconstruction error < 2%), ICA being the best. (2) To keep the pairwise topological structure, choose PCA. PCA best preserves the pairwise distance/ranking property. (3) For edge computing and IoT scenarios, choose RP or SRP if reconstruction is not required and the JL-lemma condition is met. The RP family has the best computational performance in the experiment, almost 10-100 times faster than its peers.

Computing a projection operator onto the null space of a linear imaging operator: tutorial.

Many imaging systems can be approximately described by a linear operator that maps an object property to a collection of discrete measurements. However, even in the absence of measurement noise, such operators are generally "blind" to certain components of the object, and hence information is lost in the imaging process. Mathematically, this is explained by the fact that the imaging operator can possess a null space. All objects in the null space, by definition, are mapped to a collection of identically zero measurements and are hence invisible to the imaging system. As such, characterizing the null space of an imaging operator is of fundamental importance when comparing and/or designing imaging systems. A characterization of the null space can also facilitate the design of regularization strategies for image reconstruction methods. Characterizing the null space via an associated projection operator is, in general, a computationally demanding task. In this tutorial, computational procedures for establishing projection operators that map an object to the null space of a discrete-to-discrete imaging operator are surveyed. A new machine-learning-based approach that employs a linear autoencoder is also presented. The procedures are demonstrated by use of biomedical imaging examples, and their computational complexities and memory requirements are compared.

Data analytics for cybersecurity enhancement of transformer protection

Electric power substations are experiencing an accelerated pace of digital transformation including the deployment of LAN-based IEC 61850 communication protocols that facilitate accessibility to substation data while also increasing remote access points and exposure to complex cyberattacks. In this environment, machine learning algorithms will play a vital role in cyberattack detection and mitigation and natural questions arise as to the most effective models in the context of smart grid substations. This paper compares the performance of three autoencoder-based anomaly detection systems including linear, fully connected, and convolutional autoencoders, as well as long short-term memory (LSTM) neural network for cybersecurity enhancement of transformer protection. The simulation results indicated that the LSTM model outperforms the other models for detecting cyberattacks targeting asymmetrical fault data. The linear autoencoder, fully connected autoencoder and 1D CNN further outperform the LSTM model for detecting cyberattacks targeting the symmetrical fault data.

Machine learning for fluid flow reconstruction from limited measurements

This paper investigates the use of data-driven methods for the reconstruction of unsteady fluid flow fields. The proposed framework is based on the combination of machine learning tools: dimensionality reduction to extract dominant spatial directions from data, reconstruction algorithm to recover encoded data by limited measurements and cross-validation for hyperparameter optimization. For the encoding part, linear and nonlinear extraction of patterns are considered: proper orthogonal decomposition (POD), linear autoencoder (LAE) and variational autoencoder (VAE). For the reconstruction part, regressive reconstruction (neural network, linear, support vector, gradient boosting) and library-based reconstruction are compared, each method being cross-validated to ensure good generalization on testing data. The position of sensors is optimized using an enhanced clustering algorithm. The robustness of regressive reconstructions to noise measurements is also addressed, showing the benefits of variational approaches in the reduction phase. The strategy is tested for three increasing complexity flows: 2D vortex shedding (Re=200), 2D spatial mixing layer and 3D vortex shedding (Re=20000). The results suggest that proper machine learning approaches to fluid flow data can lead to effective reconstruction models that can be used for the rapid estimation of complex flows.

Encoding-based memory for recurrent neural networks

Learning to solve sequential tasks with recurrent models requires the ability to memorize long sequences and to extract task-relevant features from them. In this paper, we study memorization from the point of view of the design and training of recurrent neural networks. We study how to maximize the short-term memory of recurrent units, an objective difficult to achieve using backpropagation. We propose a new model, the Linear Memory Network, which features an encoding-based memorization component built with a linear autoencoder for sequences. Additionally, we provide a specialized training algorithm that initializes the memory to efficiently encode the hidden activations of the network. Experimental results on synthetic and real-world datasets show that the chosen encoding mechanism is superior to static encodings such as orthogonal models and the delay line. The method also outperforms RNN and LSTM units trained using stochastic gradient descent. Experiments on symbolic music modeling show that the training algorithm specialized for the memorization component improves the final performance compared to stochastic gradient descent.

Kernelized Linear Autoencoder

This work proposes a new representation learning model called kernelized linear autoencoder. Instead of modeling non-linearity by the non-linear activation functions, we employ linear activations but account for non-linearity by the kernel trick. We propose four variants. The first one is the basic unsupervised kernelized linear autoencoder. The second one is a label consistent version for classification. The third one incorporates sparse sub-space clustering into the autoencoder formulation. Finally, we proposed a coupled version for domain adaptation problems. For each task, we compare our proposed technique with shallow and deep representation learning methods on benchmark problems. For each task, we perform significantly better than the methods compared against.

Constrained Autoencoder-Based Pulse Compressed Thermal Wave Imaging for Sub-Surface Defect Detection

Non-destructive testing & evaluation techniques play an essential role in ensuring safety of materials in operation at various industry sectors. Pulse compressed favourable thermal wave imaging is one of the widely used non-destructive testing techniques due to its excellent noise rejection capabilities. However, the high dimensional thermal imaging data needs to be encoded into lossless compressed form to highlight the hidden defects inside the materials. This paper proposes a novel constrained and regularized autoencoder based thermography approach for sub-surface defect detection in a mild steel specimen. Certain properties such as non-correlation of encoded data, weight orthogonality, and weights with unit norm length have been highlighted which are non-existent in linear autoencoders but are responsible for better defect detection inside the materials inspected by frequency modulated thermal wave imaging. Novel constraints are formulated for autoencoder cost function to incorporate these significant properties. The proposed approach is able to provide better defect detection, in terms of signal to noise ratio of defects, than linear autoencoder as well as traditional principal component thermography approach. Also, non-correlation of encoded data is found to be the most significant factor in achieving better defect detection followed by properties ensuring weight orthogonality and weights with unit norm length.

Low-rank decomposition on transformed feature maps domain for image denoising

Low-rank based models are proved outstanding for denoising on the data with strong repetitive or redundant property. However, for natural images with complex structures or rich details, the performance drops down because of the weak low-rankness of the data. A feasible solution is to transform the data into a suitable domain to further explore the underlying low-rank information. In this paper, we present a novel approach to create such a domain via a fully replicated linear autoencoder network. By applying various low-rank models to the feature maps generated by the encoder rather than the original data, and then performing inverse transformation by the decoder, their denoising performances all get enhanced. In addition, feature maps also show good sparsity, hence we introduce a new measure combining sparse and low-rank regularity, and further propose corresponding single image denoising model. Extensive experiments show the superiority of our work.

A dual linear autoencoder approach for vessel trajectory prediction using historical AIS data

Advances in artificial intelligence are driving the development of intelligent transportation systems, with the purpose of enhancing the safety and efficiency of such systems. One of the most important aspects of maritime safety is effective collision avoidance. In this study, a novel dual linear autoencoder approach is suggested to predict the future trajectory of a selected vessel. Such predictions can serve as a decision support tool to evaluate the future risk of ship collisions. Inspired by generative models, the method suggests to predict the future trajectory of a vessel based on historical AIS data. Using unsupervised learning to facilitate trajectory clustering and classification, the method utilizes a cluster of historical AIS trajectories to predict the trajectory of a selected vessel. Similar methods predict future states iteratively, where states are dependent upon the prior predictions. The method in this study, however, suggests predicting an entire trajectory, where all states are predicted jointly. Further, the method estimates a latent distribution of the possible future trajectories of the selected vessel. By sampling from this distribution, multiple trajectories are predicted. The uncertainties of the predicted vessel positions are also quantified in this study.

A Sequential Autoencoder for Teleconnection Analysis

Many aspects of the earth system are known to have preferred patterns of variability, variously known in the atmospheric sciences as modes or teleconnections. Approaches to discovering these patterns have included principal components analysis and empirical orthogonal teleconnection (EOT) analysis. The latter is very effective but is computationally intensive. Here, we present a sequential autoencoder for teleconnection analysis (SATA). Like EOT, it discovers teleconnections sequentially, with subsequent analyses being based on residual series. However, unlike EOT, SATA uses a basic linear autoencoder as the primary tool for analysis. An autoencoder is an unsupervised neural network that learns an efficient neural representation of input data. With SATA, the input is an image time series and the neural representation is a unidimensional time series. SATA then locates the 0.5% of locations with the strongest correlation with the neural representation and averages their temporal vectors to characterize the teleconnection. Evaluation of the procedure showed that it is several orders of magnitude faster than other approaches to EOT, produces teleconnection patterns that are more strongly correlated to well-known teleconnections, and is particularly effective in finding teleconnections with multiple centers of action (such as dipoles).

DeepAirSig: End-to-End Deep Learning Based in-Air Signature Verification

In-air signature verification is vital for biometric user identification in contact-less mode. The state-of-the-art methods use heuristics for signature acquisition, and provide insufficient data to train neural networks for the verification. In this article, we present a novel method for end-to-end deep learning based in-air signature verification using a depth sensor. In this regard, we propose a new medium-scale in-air signature dataset which is created using an accurate convolutional neural network (CNN) based 3D hand pose estimation algorithm. The proposed dataset offers a total of 1800 signatures collected from 40 subjects. So far, dynamic time warping (DTW) has been the most effective and commonly used method for verification. Keeping in view the significant advancement in deep learning, we present a more accurate deep learning based alternative which outperforms DTW by 67.6%. To this end, we train a personalized autoencoder to reconstruct the signature of each subject. Thereafter, the signature is verified by thresholding the reconstruction loss. We perform extensive experiments by formulating spatial and depth features of the signature in images and point clouds based representations. Moreover, for comparisons, we implement several deep learning algorithms (i.e. linear autoencoder, convolutional autoencoder, and Deep One-Class classifier). Our verification approach achieves an EER of 0.055%. The dataset is available at https://bit.ly/2mEJzOw.

From Zero-Shot Learning to Cold-Start Recommendation

Zero-shot learning (ZSL) and cold-start recommendation (CSR) are two challenging problems in computer vision and recommender system, respectively. In general, they are independently investigated in different communities. This paper, however, reveals that ZSL and CSR are two extensions of the same intension. Both of them, for instance, attempt to predict unseen classes and involve two spaces, one for direct feature representation and the other for supplementary description. Yet there is no existing approach which addresses CSR from the ZSL perspective. This work, for the first time, formulates CSR as a ZSL problem, and a tailor-made ZSL method is proposed to handle CSR. Specifically, we propose a Lowrank Linear Auto-Encoder (LLAE), which challenges three cruxes, i.e., domain shift, spurious correlations and computing efficiency, in this paper. LLAE consists of two parts, a low-rank encoder maps user behavior into user attributes and a symmetric decoder reconstructs user behavior from user attributes. Extensive experiments on both ZSL and CSR tasks verify that the proposed method is a win-win formulation, i.e., not only can CSR be handled by ZSL models with a significant performance improvement compared with several conventional state-of-the-art methods, but the consideration of CSR can benefit ZSL as well.