Autoencoder Framework Research Articles

Efficient-Deformable-DETR: Enhancing underwater small target detection in complex environments

The marine environment is facing multiple challenges, and the development of underwater small target detection technology is crucial for protecting marine ecology. This paper focuses on the problem of small target detection in complex underwater environments and proposes an improved model based on the Deformable-DETR model. By introducing global response normalization and a fully convolutional mask autoencoder framework, the model’s feature aggregation ability and detection accuracy are enhanced. At the same time, a gradient harmonization mechanism is used to solve the problem of positive and negative sample imbalance, which enhances the model’s learning performance. Experimental results show that the improved model achieves a mAP value of 84.5% on the URPC2020 dataset, an increase of 1.7% over the original model. This technical optimization not only improves the accuracy of underwater small target detection but also contributes to marine ecological monitoring, rational utilization of resources, and environmental protection, promoting the realization of green environment and sustainable development goals.

Prototype-assisted multiscale graph representation learning-based mechanical fault detection method under complex operating conditions

Effective anomaly detection and timely fault warning are essential to ensure the continuous and safe operation of mechanical equipment and to prevent equipment deterioration. In the unsupervised modeling and detection scenario, fault detection methods based on the autoencoder framework have been widely concerned and applied. Unfortunately, such methods can only be applied to specific or constant operating conditions, and their detection performance is greatly reduced due to the different data distribution in the face of complex operating conditions. Aiming at the problem of unsupervised fault detection under complex operating conditions, this article proposes a prototype-assisted multiscale graph representation learning-based mechanical fault detection method. First, the vibration data of the equipment is fed into the multiscale decomposition module (MDM) to obtain multiscale feature maps that can express rich detail information. Then, the multiscale feature maps are fed into the graph representation learning module (GRLM) to fully learn the potential relationships and interactions between different scales and provide a more comprehensive representation of the dynamic characteristics of the equipment. Finally, multiple MDMs and GRLMs are cascaded to construct a feature extractor to map the data of each operating condition to the latent space, and the proposed prototype-assisted strategy is used to determine the real-time state of the equipment. Case studies have been carried out on two different pieces of mechanical equipment. The experimental results show that the average accuracy of the proposed method is as high as 98.44% and 98.90%, respectively, and it maintains a low missed detection rate and zero false alarm rate in the two validation processes, which is more in line with the needs of engineering applications than other comparison methods.

A 1T1M-based efficient probability storage and computing cell for the VAE-SBN hybrid model

Abstract Bayesian network are crucial components for uncertainty reasoning and knowledge representation in probabilistic graphical models. As model complexity increases, traditional digital methods face challenges in efficiency and resource utilization. In this work, a novel Bayesian network optimization method based on one-transistor one-memristor (1T1M) units for storing and accessing probability values was proposed. This method leverages the sigmoid-like characteristics of 1T1M for computational optimization, integrates probability storage into sigmoid belief network (SBN), and extends to the variational autoencoder (VAE) framework. By storing and manipulating probabilities at hardware level, this approach enhances storage efficiency and access speed while enhancing computational precision and model expressiveness through the sigmoid-like space of 1T1M. Validation experiments using the MNIST digit generation task demonstrate that this method exhibits unique advantages in SBN structures, particularly in handling complex conditional probabilities and inference tasks. Compared to traditional digital implementations, the proposed method exhibits potential advantages in inference speed and hardware utilization.

Paired autoencoders for likelihood-free estimation in inverse problems

Abstract We consider the solution of nonlinear inverse problems where the forward problem is a discretization of a partial differential equation. Such problems are notoriously difficult to solve in practice and require minimizing a combination of a data-fit term and a regularization term. The main computational bottleneck of typical algorithms is the direct estimation of the data misfit. Therefore, likelihood-free approaches have become appealing alternatives. Nonetheless, difficulties in generalization and limitations in accuracy have hindered their broader utility and applicability. In this work, we use a paired autoencoder framework as a likelihood-free estimator (LFE) for inverse problems. We show that the use of such an architecture allows us to construct a solution efficiently and to overcome some known open problems when using LFEs. In particular, our framework can assess the quality of the solution and improve on it if needed. We demonstrate the viability of our approach using examples from full waveform inversion and inverse electromagnetic imaging.

SC-VAE: Sparse coding-based variational autoencoder with learned ISTA

Learning rich data representations from unlabeled data is a key challenge towards applying deep learning algorithms in downstream tasks. Several variants of variational autoencoders (VAEs) have been proposed to learn compact data representations by encoding high-dimensional data in a lower dimensional space. Two main classes of VAEs methods may be distinguished depending on the characteristics of the meta-priors that are enforced in the representation learning step. The first class of methods derives a continuous encoding by assuming a static prior distribution in the latent space. The second class of methods learns instead a discrete latent representation using vector quantization (VQ) along with a codebook. However, both classes of methods suffer from certain challenges, which may lead to suboptimal image reconstruction results. The first class suffers from posterior collapse, whereas the second class suffers from codebook collapse. To address these challenges, we introduce a new VAE variant, termed sparse coding-based VAE with learned ISTA (SC-VAE), which integrates sparse coding within variational autoencoder framework. The proposed method learns sparse data representations that consist of a linear combination of a small number of predetermined orthogonal atoms. The sparse coding problem is solved using a learnable version of the iterative shrinkage thresholding algorithm (ISTA). Experiments on two image datasets demonstrate that our model achieves improved image reconstruction results compared to state-of-the-art methods. Moreover, we demonstrate that the use of learned sparse code vectors allows us to perform downstream tasks like image generation and unsupervised image segmentation through clustering image patches. The code is available at https://github.com/sotiraslab/SC-VAE.

Toward Domain Adaptive Learning-Based Variation Autoencoder Emotional Analysis in English Teaching

Speech emotion analysis plays an important role in English teaching by analyzing the reading state of students. Teachers can dynamically adjust the teaching content according to the emotional feedback of students and improve the teaching quality of the school. Due to unstable student emotions and background noise, the accuracy of speech emotion recognition is constrained. Although multimodal data can alleviate the deficiency of a single modality, collecting and annotating multimodal samples requires a significant amount of resources. To resolve this issue, this paper proposes a novel multimodal sentiment analysis framework based on domain adaptive learning mechanisms to assist English teaching. We construct a novel multi-task variation autoencoder framework in which we simultaneously complete reconstruction and classification tasks. To improve speech emotion recognition performance, we introduce domain adaptive learning based on the Wasserstein distance between two variational hidden layers from the video domain (source domain) and speech domain (target domain). To validate the effectiveness of our proposed model, we conducted extensive comparative experiments on two public datasets and a self-built English oral dataset. All experimental results indicate that domain adaptation learning mechanisms can effectively improve the recognition performance of the target domain. On the self-built dataset for English teaching, the proposed model achieves higher performance compared to other deep models.

Multimodal Trajectory Prediction for Diverse Vehicle Types in Autonomous Driving with Heterogeneous Data and Physical Constraints.

The accurate prediction of vehicle behavior is crucial for autonomous driving systems, impacting their safety and efficiency in complex urban environments. To address the challenge of multi-agent trajectory prediction, we propose a novel model integrating multiple input modalities, including historical trajectories, map data, vehicle features, and interaction information. Our approach employs a Conditional Variational Autoencoder (CVAE) framework with a decoder that predicts control actions using the Gaussian Mixture Model (GMM) and then converts these actions into dynamically feasible trajectories through a bicycle model. Evaluated on the nuScenes dataset, the model achieves great performance across key metrics, including minADE5 of 1.26 and minFDE5 of 2.85, demonstrating robust performance across various vehicle types and prediction horizons. These results indicate that integrating multiple data sources, physical models, and probabilistic methods significantly improves trajectory prediction accuracy and reliability for autonomous driving. Our approach generates diverse yet realistic predictions, capturing the multimodal nature of future outcomes while adhering to Physical Constraints and vehicle dynamics.

Expressive 3D Facial Animation Generation Based on Local-to-Global Latent Diffusion.

3D Facial animations, crucial to augmented and mixed reality digital media, have evolved from mere aesthetic elements to potent storytelling media. Despite considerable progress in facial animation of neutral emotions, existing methods still struggle to capture the authenticity of emotions. This paper introduces a novel approach to capture fine facial expressions and generate facial animations using audio synchronization. Our method consists of two key components: First, the Local-to-global Latent Diffusion Model (LG-LDM) tailored for authentic facial expressions, which can integrate audio, time step, facial expressions, and other conditions towards possible encoding of emotionally rich yet latent features in response to possibly noisy raw audio signals. The core of LG-LDM is our carefully designed Facial Denoiser Model (FDM) for aligning the local-to-global animation feature with audio. Second, we redesign an Emotion-centric Vector Quantized-Variational AutoEncoder framework (EVQ-VAE) to finely decode the subtle differences under different emotions and reconstruct the final 3D facial geometry. Our work significantly contributes to the key challenges of emotionally realistic 3D facial animation for audio synchronization and enhances the immersive experience and emotional depth in augmented and mixed reality applications. We provide a reproducibility kit including our code, dataset, and detailed instructions for running the experiments. This kit is available at https://github.com/wangxuanx/Face-Diffusion-Model.

Parameter-adaptive variational autoencoder for linear/nonlinear blind source separation

AbstractBlind source separation (BSS) serves as an important technique in the field of structural health monitoring (SHM), particularly for solving modal decomposition tasks. This study proposes a novel approach to both linear and nonlinear BSS problems in the Variational Autoencoder (VAE) framework, where the encoding and decoding processes of VAE are interpreted as procedures for inferring sources from observations and remixing these sources, respectively. In this way, the distribution of latent variables inferred by VAE is equivalent to the distribution of sources. We make improvements to the vanilla VAE to augment its effectiveness for BSS. First, we substitute standard normal distributions with trainable Gaussian processes (GP) as priors for latent variables and implement an exponential function as the activation function for adaptive parameters in the GP kernel functions. While the form of the priors is set as GP, the parameters of their kernel functions are not fixed but automatically converge to suitable values during the model training process. Additionally, a hyperparameter is introduced to balance the terms in the loss function. The proposed method is referred to as parameter-adaptive VAE (PAVAE). Then, upon different assumptions of the variances of sources, the proposed PAVAE is branched into two types: homoscedastic PAVAE (Ho-PAVAE) and heteroscedastic PAVAE (He-PAVAE). Through numerical and laboratory experiments, we demonstrate the effectiveness of the proposed method in solving BSS problems and their potential to underpin future research in SHM.

Novel sparse auto-encoder framework with pseudo-labeled reinforcement for cross domain fault diagnosis with imbalanced samples

Abstract By applying diagnostic expertise from one area to another that is closely related, transfer fault diagnosis is an efficient strategy for guaranteeing the secure and dependable operation of mechanical equipment. However, the majority of rotating machinery monitoring data are gathered in industrial applications under normal operating settings, which leads to an imbalance between positive and negative samples. Consequently, performing high-precision fault diagnosis with imbalanced data and different working conditions becomes challenging due to the escalating difficulty and cost of acquiring labeled fault samples. For cross domain fault diagnosis, an enhanced deep transfer sparse auto-encoder (SAE) framework is provided. This approach leverages a SAE to delve deeper into fault features within imbalanced data and reconstruct the samples accordingly. Furthermore, the research proposes a feature domain penalty term to facilitate cross-domain training by aligning the distribution of the source and target domain data which can reduce data distribution deviation. A pseudo-labeled reinforcement training method is presented to further improve cross-domain classification accuracy with imbalanced samples. Extensive experiments were conducted on two datasets to assess the proposed method. The results were compared with other algorithms, demonstrating the effectiveness and superiority of the proposed approach.

Accurate prediction of drug-target interactions in Chinese and western medicine by the CWI-DTI model

Accurate prediction of drug-target interactions (DTIs) is crucial for advancing drug discovery and repurposing. Computational methods have significantly improved the efficiency of experimental predictions for drug-target interactions in Western medicine. However, accurately predicting the complex relationships between Chinese medicine ingredients and targets remains a formidable challenge due to the vast number and high heterogeneity of these ingredients. In this study, we introduce the CWI-DTI method, which achieves high-accuracy prediction of DTIs using a large dataset of interactive relationships of drug ingredients or candidate targets. Moreover, we present a novel dataset to evaluate the prediction accuracy of both Chinese and Western medicine. Through meticulous collection and preprocessing of data on ingredients and targets, we employ an innovative autoencoder framework to fuse multiple drug (target) topological similarity matrices. Additionally, we employ denoising blocks, sparse blocks, and stacked blocks to extract crucial features from the similarity matrix, reducing noise and enhancing accuracy across diverse datasets. Our results indicate that the CWI-DTI model shows improved performance compared to several existing state-of-the-art methods on the datasets tested in both Western and Chinese medicine databases. The findings of this study hold immense promise for advancing DTI prediction in Chinese and Western medicine, thus fostering more efficient drug discovery and repurposing endeavors. Our model is available at https://github.com/WANG-BIN-LAB/CWIDTI.

MKAN-MMI: empowering traditional medicine-microbe interaction prediction with masked graph autoencoders and KANs.

The growing microbial resistance to traditional medicines necessitates in-depth analysis of medicine-microbe interactions (MMIs) to develop new therapeutic strategies. Widely used artificial intelligence models are limited by sparse observational data and prevalent noise, leading to over-reliance on specific data for feature extraction and reduced generalization ability. To address these limitations, we integrate Kolmogorov-Arnold Networks (KANs), independent subspaces, and collaborative decoding techniques into the masked graph autoencoder (Mask GAE) framework, creating an innovative MMI prediction model with enhanced accuracy, generalization, and interpretability. First, we apply Bernoulli distribution to randomly mask parts of the medicine-microbe graph, advancing self-supervised training and reducing noise impact. Additionally, the independent subspace technique enables graph neural networks (GNNs) to learn weights independently across different feature subspaces, enhancing feature expression. Fusing the multi-layer outputs of GNNs effectively reduces information loss caused by masking. Moreover, using KANs for advanced nonlinear mapping enhances the learnability and interpretability of weights, deepening the understanding of complex MMIs. These measures significantly enhanced the accuracy, generalization, and interpretability of our model in MMI prediction tasks. We validated our model on three public datasets with results showing that our model outperformed existing leading models. The relevant data and code are publicly accessible at: https://github.com/zhuoninnin1992/MKAN-MMI.

GeomorPM: a geomorphic pretrained model integrating convolution and Transformer architectures based on DEM data

As the domain of artificial intelligence has advanced, the integration of deep learning techniques into terrain and landform analysis has become more prevalent. Nevertheless, many existing methods are fully supervised and designed for specific tasks; thus, their transferability is limited and massive annotated samples are required. This study introduces a geomorphic pretrained model (GeomorPM) capable of performing multiple tasks. First, an architecture was designed that combined a convolution-based Vector Quantised-Variational Autoencoder (VQVAE) with a Transformer-based masked autoencoder (MAE) framework, allowing it to autonomously learn local details and global patterns from large-scale digital elevation model (DEM) data. Subsequently, GeomorPM, based on the VQMAE architecture, was pretrained on massive DEM data and fine-tuned for three specific tasks: DEM void filling, DEM superresolution, and landform classification. GeomorPM outperformed the traditional and other deep learning methods in all three tasks, demonstrating the superior learning ability and transferability of the model. This study provides a practical framework for developing pretrained models based on DEMs that can be expanded to other continuous geoscientific data.

Foundation models for cardiovascular disease detection via biosignals from digital stethoscopes

Auscultation of the heart and the electrocardiogram (ECG) are two central components of the cardiac exam. Recent innovations of the stethoscope have enabled the simultaneous acquisition of a high-quality digital acoustic signal and ECG. We present foundation models trained on phonocardiogram (PCG) and ECG data collected from digital stethoscopes during routine clinical practice. We show that these foundation models that are pre-trained on large unlabeled datasets in a self-supervised manner can be fine-tuned for a variety of cardiovascular disease detection tasks. This is the first study that builds foundation models specifically for synchronously captured PCG and ECG data. Our approach is based on the recently developed masked autoencoder framework which we extend to handle multiple signals that are synchronously captured. This paradigm makes it possible to use large capacity models leading to superior performance even though the size of datasets with medical label annotations may be limited.

Identification of dynamic networks community by fusing deep learning and evolutionary clustering

Community detection is a critical component of network analysis and a hot topic in social computing. Detecting community structure in dynamic networks has important theoretical and practical implications for understanding the intrinsic function of networks and predicting network behavior. However, the majority of existing dynamic community detection methods adopt shallow models, which have limited ability to excavate complex non-linear structures and tend to generate undesirable community structures. In order to obtain an accurate and robust community structure in dynamic networks, we are inspired by network representation learning and utilize the deep learning to detect evolving communities in dynamic networks. In this paper, we propose a novel dynamic community detection method by fusing Deep Learning and Evolutionary Clustering (DLEC). This work attempts to combine deep learning and evolutionary clustering into a unified framework. First, we propose a matrix construction strategy to fully reveal the inherent community structures via the underlying community memberships. Then, we develop a novel multi-layer deep autoencoder framework that consists of multiple non-linear functions to extract the latent deep representation of the dynamic network. Based on the evolutionary clustering framework, a graph regularization term is introduced to ensure the smoothness of the community evolution. Finally, we employ the K-means clustering algorithm on the low-dimensional network space to obtain the community structure. Extensive experimental results on synthetic and real-world networks show that the proposed DLEC algorithm can effectively detect high-quality communities in dynamic networks.

Sparse-Coding Variational Autoencoders.

The sparse coding model posits that the visual system has evolved to efficiently code natural stimuli using a sparse set of features from an overcomplete dictionary. The original sparse coding model suffered from two key limitations; however: (1) computing the neural response to an image patch required minimizing a nonlinear objective function via recurrent dynamics and (2) fitting relied on approximate inference methods that ignored uncertainty. Although subsequent work has developed several methods to overcome these obstacles, we propose a novel solution inspired by the variational autoencoder (VAE) framework. We introduce the sparse coding variational autoencoder (SVAE), which augments the sparse coding model with a probabilistic recognition model parameterized by a deep neural network. This recognition model provides a neurally plausible feedforward implementation for the mapping from image patches to neural activities and enables a principled method for fitting the sparse coding model to data via maximization of the evidence lower bound (ELBO). The SVAE differs from standard VAEs in three key respects: the latent representation is overcomplete (there are more latent dimensions than image pixels), the prior is sparse or heavy-tailed instead of gaussian, and the decoder network is a linear projection instead of a deep network. We fit the SVAE to natural image data under different assumed prior distributions and show that it obtains higher test performance than previous fitting methods. Finally, we examine the response properties of the recognition network and show that it captures important nonlinear properties of neurons in the early visual pathway.

Multi-Scale Information Fusion and Decoupled Representation Learning for Robust Microbe-Disease Interaction Prediction

Research indicates that microbe activity within the human body significantly influences health by being closely linked to various diseases. Accurately predicting microbe-disease interactions (MDIs) offers critical insights for disease intervention and pharmaceutical research. Current advanced AI-based technologies automatically generate robust representations of microbes and diseases, enabling effective MDI predictions. However, these models continue to face significant challenges. A major issue is their reliance on complex feature extractors and classifiers, which substantially diminishes the models’ generalizability. To address this, we introduce a novel graph autoencoder framework that utilizes decoupled representation learning and multi-scale information fusion strategies to efficiently infer potential MDIs. Initially, we randomly mask portions of the input microbe-disease graph based on Bernoulli distribution to boost self-supervised training and minimize noise-related performance degradation. Secondly, we employ decoupled representation learning technology, compelling the graph neural network (GNN) to independently learn the weights for each feature subspace, thus enhancing its expressive power. Finally, we implement multi-scale information fusion technology to amalgamate the multi-layer outputs of GNN, reducing information loss due to occlusion. Extensive experiments on public datasets demonstrate that our model significantly surpasses existing top MDI prediction models. This indicates that our model can accurately predict unknown MDIs and is likely to aid in disease discovery and precision pharmaceutical research. Code and data are accessible at: https://github.com/shmildsj/MDI-IFDRL.

SAE-Impute: imputation for single-cell data via subspace regression and auto-encoders

BackgroundSingle-cell RNA sequencing (scRNA-seq) technology has emerged as a crucial tool for studying cellular heterogeneity. However, dropouts are inherent to the sequencing process, known as dropout events, posing challenges in downstream analysis and interpretation. Imputing dropout data becomes a critical concern in scRNA-seq data analysis. Present imputation methods predominantly rely on statistical or machine learning approaches, often overlooking inter-sample correlations.ResultsTo address this limitation, We introduced SAE-Impute, a new computational method for imputing single-cell data by combining subspace regression and auto-encoders for enhancing the accuracy and reliability of the imputation process. Specifically, SAE-Impute assesses sample correlations via subspace regression, predicts potential dropout values, and then leverages these predictions within an autoencoder framework for interpolation. To validate the performance of SAE-Impute, we systematically conducted experiments on both simulated and real scRNA-seq datasets. These results highlight that SAE-Impute effectively reduces false negative signals in single-cell data and enhances the retrieval of dropout values, gene-gene and cell-cell correlations. Finally, We also conducted several downstream analyses on the imputed single-cell RNA sequencing (scRNA-seq) data, including the identification of differential gene expression, cell clustering and visualization, and cell trajectory construction.ConclusionsThese results once again demonstrate that SAE-Impute is able to effectively reduce the droupouts in single-cell dataset, thereby improving the functional interpretability of the data.

Meta Koopman decomposition for time series forecasting under temporal distribution shifts

Time series forecasting facilitates various real-world applications and has attracted great research interests. In real-world scenarios, time series forecasting models confront a fundamental issue of temporal distribution shifts (TDS), i.e., the statistical properties of time series are evolving over time. In this paper, we utilize Koopman theory to address temporal distribution shifts. Koopman theory states any time series can be mapped into a Koopman space by proper measurement functions with their dynamics modeled by the infinite-dimensional linear Koopman operator. Therefore, time series under different distributions can be modeled by different Koopman operators. Considering the linearity of Koopman operators, the Koopman operators for representing time series under different distributions can be decomposed as a linear combination of a set of Koopman operators, which we termed as meta Koopman operators. We further theoretically show the infinite-dimensional Koopman operators can be approximated by finite matrix multiplications and the meta Koopman operators are equivalent to a set of matrices. Based on the analysis, we propose an auto-encoder framework for implementing the meta Koopman decomposition of time series, which is theoretically capable of handling TDS. Extensive experiments conducted on four real-world time series datasets demonstrate the superiority of the proposed model in tackling temporal distribution shifts.

Differentially private recommender framework with Dual semi-Autoencoder

To provide much better recommendation service, traditional recommender systems collect a large amount of user information, which, if obtained and analyzed maliciously, can cause incalculable damage to users. Therefore, differential privacy techniques, such as noise injection, have been widely introduced into recommender systems to safeguard users’ sensitive information. However, the introduction of privacy noise will lead to a degradation in recommendation quality. Hence, it is pragmatic to design a system that can furnish high quality recommendation and ensure privacy guarantee. In this article, we design a novel Differentially private recommender system with Dual Semi-Autoencoder recommender framework referred to as DP-DAE, which aims to improve the quality of recommendation while protecting user privacy. Specifically, DP-DAE is a hybrid framework of dual autoencoder and matrix factorization, which can effectively reduce data dimensionality to extract intricate features. In practice, to prevent potential privacy leaks, the differential privacy mechanism is incorporated into DP-DAE via introducing extra noise. Moreover, theoretical analysis certificates that DP-DAE satisfies ϵ-differential privacy. We do the experimental evaluation for DP-DAE over FilmTrust, Movielens-1M and Movielens-10M. The experimental results indicate that DP-DAE can provide privacy protection as well as high performance in recommendation tasks.